Abstract
Background:
Group treatment is delivered in youth correctional facilities, yet groups may be iatrogenic. Few measures with demonstrated psychometric properties exist to track behaviors of individuals during groups. We assessed psychometrics for the Group Process-Individual Level measure (GP-IL) of group treatment.
Methods:
N=152 teens were randomized to 1 of 2 groups (10 sessions each). Adolescents, counselors, and observers rated teen behaviors at sessions 3 and 10. GP-IL assesses reinforcement for deviancy and positive behaviors, member rejection, and counselor connection and praise.
Results:
Internal consistency and 1-month stability were demonstrated. Concurrent validity is supported by correlations with measures expected to be associated with group behavior (e.g., coping skills). Counselors and observers rated more deviancy during interactive skills-building groups versus didactic psycho-educational groups (p≤ .005). Scales evidenced incremental validity.
Conclusions:
GP-IL offers a sound method of tracking adolescent behaviors for professionals working with groups. Counselors ratings were most reliable and valid overall.
Keywords: Group assessment, delinquency, substance treatment
INTRODUCTION
Approximately half of juvenile detainees meet criteria for substance use disorder.1Of serious adolescent offenders, 57% reported smoking marijuana in the past 6 months, averaging 1-3 times/week; 40% reported consuming alcohol in the past 6 months, averaging 1-3 times/month; and 27% reported using other illicit substances an average of 1-2 times in the past 6 months.2
Psycho-education and other substance abuse treatments in correctional facilities are often provided in group format, and frequently include components of cognitive-behavioral therapy (CBT) and 12-step approaches.3Group treatment has been related to iatrogenic effects with at-risk adolescents,4-6especiallyfor adolescents reporting low levels of delinquency prior to group CBT.7In contrast, others have found that among adolescents, group CBTs are consistently associated with reduced substance use,8and areas efficacious as other interventions in decreasing substance use and delinquency.9
One difficulty in studying adolescent group-based treatment is the lack of user-friendly, validated instruments, coded from multiple vantage points (adolescent, counselor, observer), particularly for offenders. A related difficulty is the use of different instruments across different studies that measure different constructs.8 Studieson efficacy of group interventions rarely include a reliable and valid measure of group process.
One exception is an observational measure of group process (OGP) to code children attending 12 sessions of CBT.5 Trained undergraduates coded the first and last 15-minutes of group, along with a 15-minute break. A 4-point Likert scale was used (0 = no examples, was not observed, to 4 = multiple examples or one salient event [very true for client]) to code deviancy training, positive group involvement, peer rejection, and therapist praise for positive behavior. The average number of older peer counselors with whom the client seemed to have a positive relationship was recorded. Across sessions, excellent split-half reliabilities and internal consistencies were found for three scales (none were reported for connectedness or therapist praise). Although no formal validity data were presented, the reinforcement for deviancy score predicted cigarette use and delinquency at 3-year follow-up, connectedness to peer counselor predicted better outcomes, and peer rejection score predicted better smoking outcome.
The purpose of this study is to examine the reliability and validity of an adapted version of the observational measure of group process as developed by Dishion et al.5 The OGP measures various elements of group process thought to be important to outcome (adolescent relationship to peers/counselor, counselor behavior, adolescent negative/positive behaviors), and it is relatively brief and easily scored. However, because the original measure only uses trained observers, we developed versions to be completed by adolescent and therapist to determine if these may provide a cost-efficient means of obtaining the relevant information. This expanded version is called the Group Process-Individual Level measure (GP-IL).
METHODS
Participants
The site was a state juvenile correctional facility in the Northeast. All procedures received Institutional Review Board approval. Immediately after adjudication eligible adolescents were recruited with consent from legal guardians, and assent from adolescents (≥18years provided consent). This study was part of a larger randomized clinical trial testing 2 behavioral interventions for substance use, to be reported in full elsewhere. Guardians and adolescents were informed that all information was confidential, except for plans to escape or hurt self/others, or reports of child abuse. Adolescents were included in the study if they met the following screening criteria:(a) ages 14-19 years; (b) sentenced for 4-12 months; and (c) in the year before incarceration they used marijuana or drank at least monthly, or they drank heavily (≥5standard drinks for boys, ≥4 for girls) at least once; or they used marijuana or drank in the 4weeks before either the offense for which they were incarcerated, or before they were incarcerated.
Of 1,109 adolescents screened for the study, approximately 152 met screening criteria and completed the consent procedure. Of those 152 enrolled at baseline, 134 and 129 completed the first and second in-facility follow-ups, respectively. Reasons for not completing these follow-ups were change in sentence length (n=6) and lack of interest in completing study (n = 17).
Of the baseline sample (N =152),38.8%were Hispanic, 36.8% African American, 30.9%White, 7.9% Native American, 5.3% Pacific Islander, 4.6% Asian American, and 14.5% self-identified as other. Average age was 16.9 years (standard deviation, SD =1.09), 88.2% were boys, and they had been detained or incarcerated M = 2.54(SD = 2.41) times before the current incarceration. In the previous year, 27.7% and 59.9% met criteria for alcohol and marijuana dependence, respectively. Those included in the study (n = 129) at furthest follow-up were compared to those not included (n = 23) on important variables such as substance use disorder, gender, age, ethnicity/race and conduct disorder symptom count; no differences were found.
Procedures
Facility description
Charges range from simple truancy to violent offenses. Adolescents receive group and individual treatment on a variety of topics (sex-offending, anger management, etc.) and each has an assigned social worker. Comprehensive health, education and other services are provided.
Assessments
60- to 90-min interviews were conducted by a trained bachelor’s, master’s or doctoral-level Research Assistant (RA). RAs had about 20 hours of training; senior level staff conducted in vivo observations to ensure proper assessment delivery. Following training, weekly supervision was provided by a PhD-level staff member; all assessment data were reviewed by a senior-level staff member. Assessments occurred at baseline (shortly after adjudication), and after sessions 3 and 10. Adolescents received $35 and snacks for participation.
Interventions
Following baseline assessment, adolescents were randomized to 2 sessions of individually delivered Motivational Interviewing followed by 10 group sessions of Cognitive Behavioral Therapy (MI/CBT),10-12 or 2 sessions of Relaxation Training followed by 10 group sessions of Substance Education and Twelve-Step Introduction (RT/SET).13 Treatment effects on process and outcome will be reported elsewhere; the individual treatment is not relevant to the present study. All interventions were manualized, including fidelity procedures. Group sessions were about 75 minutes each, gender segregated, used rolling admission, and occurred 1-3 times/week with about 3 participants/group session. On average, adolescents received 8 group sessions over 6.5 weeks.
Counselors had about 160 hours of manualized training, with weekly supervision. Training consisted of readings and role-plays with feedback. Intervention files were reviewed by a clinical psychologist. Counselors were 1 man and 5 women; all 6 were Caucasian; 4 had a master's, 1 had a bachelor's and 1 had a doctoral degree. Each counselor conducted both intervention types. All sessions were recorded, and fidelity coding was conducted (data not reported here). Any counselors falling below fidelity received added individual supervision and tracking until fidelity again met acceptable levels.
Measures
A Background Questionnaire was used at baseline to record socio-demographic information including age, gender and race. Composite International Diagnostic Interview short-form (CIDI-SF)14alcohol and marijuana dependence modules were administered at baseline. Alcohol and Marijuana Ladders (AL, ML),15,16 provided at baseline, assessed readiness to change. Brief Situational Confidence Questionnaire-Alcohol and Marijuana (BSCQ-A, M),17provided at baseline and follow-up,yields total confidence scores to resist alcohol and marijuana, separately. Center for Epidemiological Studies Depression Scale (CES-D)18 was provided at baseline. Treatment Participation Questionnaire (TPQ)19 has 3 versions completed by the adolescent, counselor and social worker. The measure consists of positive (sample item: “I think a lot about the good and bad things about substance use”) and negative (“I like to joke in treatment when they begin discussing substance use”) substance treatment engagement scales. The adolescent version was provided at baseline, and all 3 versions were provided at the 3rd and 10th session assessments. Adolescent Relapse Coping Questionnaire (ARC-Q),20-22 administered at baseline and at 3rd and 10th session assessments, provides scores for cognitive and behavior problem solving, self-critical thinking, abstinence-focused coping, and coping self-efficacy. Delinquent Activities Scale (DAS),23,24 administered at baseline, was utilized to obtain conduct disorder symptom count.
Group Process-Individual Level measure (GP-GL) consists of scales measuring Reinforcement for Deviance (Dev; 6 items), Connection to Counselor (Con; Yes/No rating for each counselor, 1 item), Positive Group Involvement (Pos; 4 items), Peer Rejection (Rej; 3 items), and Counselor Praise for Positive Behavior (Pra; 3 items). Items are rated on a Likert scale (0 = “No examples, was not observed” to 4 = “Multiple examples or one clear event [very true for teen]”), with averages calculated for each scale. This measure is completed at the 3rd and 10th session assessments by adolescents, counselors, and observers. For this study, the observer version is coded in 10 minute segments for each of 3 blocks at the beginning, middle and end of a group session. Observer training consisted of reading the coding manual, coding mock sessions, feedback by senior staff members, and meeting coding criteria (all ratings within ±1) on tapes coded consistently by 2 PhD-level staff members. One master's-level, two bachelor's-level, and two undergraduate RAs performed ratings; all were Caucasian; one was a man and four were women. Adolescents (20%) were randomly chosen for double-coding of observer data. Monitoring of data for consistency was conducted by a senior level staff member every four months on average; feedback was provided if criteria were not maintained on 75% or more of items. Adolescent and counselor forms are identical to the observer version with some wording modification for adolescents. Immediately following the 3rd and 10th sessions, an RA assisted the adolescent in completing the form and the counselor independently completed his/her form.
RESULTS
Data were checked for distributional assumptions and transformed as needed. Peer rejection rated by adolescents (Rej-A) and deviancy training as rated by observers (Dev-O) were transformed so that the furthest three outliers were assigned values of 3, 2 and 1 unit larger than 3 standard deviations above the mean.25Peer rejection as rated by observers (Rej-O) and its items were poorly distributed; therefore, this scale was dichotomized into presence (1) or absence (0) of peer rejection.
Internal Consistency and Descriptives
Cronbach α was calculated for observer, adolescent and counselor versions at sessions 3 and 10on each scale (for observers, α was calculated per block). Item deletions were examined to improve α with the constraint that items on a particular scale remain consistent across adolescent, counselor and observer versions. For peer rejection rated by observers (Rej-O), internal consistency could not be calculated because data would not conform to distributional assumptions. Therefore, Rej-O is more appropriately termed an index and further analyses involving Rej-O utilized the dichotomized indicator. Descriptive statistics for scales and indices and α are found in Table 1.
TABLE 1.
Description of Scales and Indices
| Scale (Number of items) | Observera | Counselor | Adolescent | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| ||||||||||||
| 3rd | 10th | 3rd | 10th | 3rd | 10th | |||||||
|
| ||||||||||||
| n | 127 | 88 | 138 | 129 | 134 | 129 | ||||||
|
| ||||||||||||
| α | M(SD) or % yes |
α | M(SD) or % yes |
α | M(SD) or % yes |
α | M(SD) or % yes |
α | M(SD | α | M(SD) or % yes |
|
| Deviancy (4) | .85 | .31(.43) | .88 | .30(.39) | .93 | 1.12(.80) | .96 | 1.31(.87) | .79 | .78(.83) | .79 | .78(.79) |
| Positive Involvement (3) | .69 | 1.94(.58) | .57 | 1.76(.60) | .86 | 2.88(.56) | .87 | 2.89(.50) | .73 | 3.18(.79) | .64 | 3.16(.77) |
| Peer Rejection (3) | -b | 80.3% | -b | 79.5% | .78 | .33(.44) | .83 | .44(.52) | .56 | .16(.37) | .63 | .13(.29) |
| Counselor Praise (3) | .45 | .41(.42) | .58 | .31(.40) | .68 | 2.46(.39) | .61 | 2.51(.34) | .51 | 3.05(.84) | .69 | 3.31(.76) |
| Counselor Connection (1) | - | 65.4% | - | 63.6% | - | 84.8% | - | 86.8% | - | 97.8% | - | 98.4% |
Median α presented for 3 blocks and statistic provided for the 3 blocks combined (e.g., .31 is the average at session 3 across the 3 blocks for Deviancy).
Distributions on this measure were poor and αs could not be meaningfully calculated.
n = number of adolescents coded.
M = mean; SD = standard deviation; % = percent. % Yes is percent of adolescents for whom the index was endorsed as occurring (e.g., 65.4% of adolescents rated by observers as being connected to counselors on the Connection index at session 3). Dashed line (-) indicates the statistic does not apply.
Correlational Analyses
Table 2 demonstrates correlations among scales as might be expected; for example, r=.37 (p≤.001) between observers and counselor deviancy ratings, and similarly, r=.44 (p≤.001) between observer ratings of counselor praise and positive teen behaviors. Table 3 indicates that although baseline constructs do not show overwhelming association with group processes, treatment engagement and skills to be disseminated during sessions are associated with group processes (e.g., adolescent ratings of counselor praise are associated with problem-solving skills; r=.36, p≤.001). Table 4 demonstrates group processes at session 3 are associated with later treatment engagement and abstinence-focused coping at session 10; however, session 3 group processes are not necessarily associated with later confidence or efficacy to reduce use.
TABLE 2.
Correlations among Scales and Indices at 3rd Session Assessment
| Dev |
Pos |
Rej |
Pra |
Cona |
|||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A | C | O | A | C | O | A | C | Oa | A | C | O | A | C | O | |
| Dev | |||||||||||||||
| A | - | .25 | X | −.23 | X | X | X | X | X | −.14 | X | X | X | X | X |
| p≤ | .002 | .004 | .049 | ||||||||||||
| C | - | .37* | −.18 | X | X | X | .41* | X | X | X | X | X | X | X | |
| p≤ | .001 | .019 | .001 | ||||||||||||
| O | - | X | X | .17 | X | .19 | X | .18 | X | .18 | X | −.22 | X | ||
| p≤ | .027 | .019 | .025 | .002 | .008 | ||||||||||
| Pos | |||||||||||||||
| A | - | X | X | X | X | X | .35* | .16 | X | X | −.21 | X | |||
| p≤ | .001 | .034 | .008 | ||||||||||||
| C | - | .43* | X | X | X | .20 | .32* | .17 | X | .45* | .15 | ||||
| p≤ | .001 | .012 | .001 | .030 | .001 | .049 | |||||||||
| O | - | X | X | X | .20 | X | .44* | X | X | .41* | |||||
| p≤ | .013 | .001 | .001 | ||||||||||||
| Rej | |||||||||||||||
| A | - | X | X | X | X | X | X | X | X | ||||||
| p≤ | |||||||||||||||
| C | - | X | X | X | X | X | .21 | x | |||||||
| p≤ | .006 | ||||||||||||||
| O | - | X | X | X | X | X | .28 | ||||||||
| p≤ | .001 | ||||||||||||||
| Pra | |||||||||||||||
| A | - | .27* | X | .25 | X | .33* | |||||||||
| p≤ | .001 | .002 | .001 | ||||||||||||
| C | - | X | .18 | X | X | ||||||||||
| p≤ | .021 | ||||||||||||||
| O | - | X | .16 | X | |||||||||||
| p≤ | .040 | ||||||||||||||
| Cona | |||||||||||||||
| A | - | xb | .22b | ||||||||||||
| p≤ | .006 | ||||||||||||||
| C | - | xb | |||||||||||||
| p≤ | |||||||||||||||
| O | - | ||||||||||||||
| p≤ | |||||||||||||||
Point-biserial correlation between continuous and dichotomous variables.
Phi coefficient between 2 dichotmous variables.
Significant after Bonferroni adjustment for (15×15)/2 comparisons (.05/113).
A = adolescent, C = counselor, O = observer. Dev = Deviancy, Con = Counselor Connection, Pos = Positive Behavior, Rej = Peer Rejection, Pra = Counselor Praised Behavior. X = non-significant.
TABLE 3.
Correlations of Group Process with Relevant Constructs
| Baseline |
3rd Session Assessment |
|||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TPQ |
ARC-Q |
|||||||||||||
| A | C | SW | ||||||||||||
| CD | AL | ML | D | P | N | P | N | P | N | SE | PS | SC | AB | |
| Dev | ||||||||||||||
| A | .25 | X | X | X | X | .29* | X | .28 | X | X | X | X | X | X |
| p≤ | .002 | .001 | .001 | |||||||||||
| C | X | −.16 | X | X | −.27 | .37* | −.25 | .80* | X | .18 | −.17 | X | X | −.20 |
| p≤ | .031 | .001 | .001 | .002 | .001 | .016 | .027 | .010 | ||||||
| O | .22 | −.16 | X | X | X | X | X | .33* | X | X | X | X | X | X |
| p≤ | .007 | .034 | .001 | |||||||||||
| Pos | ||||||||||||||
| A | X | X | X | .19 | .30* | X | .21 | X | X | X | .17 | .22 | X | .24 |
| p≤ | .013 | .001 | .007 | .024 | .005 | .002 | ||||||||
| C | X | X | X | X | .22 | X | .62* | −.26 | .22 | −.16 | X | X | X | X |
| p≤ | .005 | .001 | .001 | .006 | .029 | |||||||||
| O | X | X | −.18 | .25 | .22 | X | .19 | X | X | X | .25 | X | X | X |
| p≤ | .002 | .002 | .007 | .016 | .003 | |||||||||
| Rej | ||||||||||||||
| A | X | X | X | X | X | .14 | X | X | X | X | X | X | X | X |
| p≤ | .048 | |||||||||||||
| C | X | X | X | −.15 | −.18 | X | X | .25 | X | X | −.24 | X | X | X |
| p≤ | .037 | .017 | .002 | .003 | ||||||||||
| O | X | X | X | X | X | X | X | .15 | X | X | X | X | X | X |
| p≤ | .047 | |||||||||||||
| Pra | ||||||||||||||
| A | X | X | X | .20 | .46* | X | .26 | X | .16 | X | .19 | .36* | .16 | .31* |
| p≤ | .011 | .001 | .001 | .034 | .015 | .001 | .030 | .001 | ||||||
| C | .18 | X | X | X | X | X | .44* | X | .22 | X | X | X | −.14 | X |
| p≤ | .016 | .001 | .005 | .048 | ||||||||||
| O | X | X | X | X | X | −.23 | X | X | X | X | X | X | X | X |
| p≤ | .005 | |||||||||||||
| Cona | ||||||||||||||
| A | X | X | X | X | .19 | X | .28 | X | X | X | .17 | .22 | .14 | .19 |
| p≤ | .014 | .001 | .023 | .005 | .049 | .014 | ||||||||
| C | X | −.14 | X | −.19 | X | X | .48* | X | X | X | −.22 | X | X | X |
| p≤ | .048 | .015 | .001 | .005 | ||||||||||
| O | X | X | X | X | .25 | X | X | X | X | X | .21 | X | X | X |
| p≤ | .003 | .009 | ||||||||||||
A = adolescent, C = counselor, O = observer. Dev = Deviancy, Con = Counselor Connection, Pos = Positive Behavior, Rej = Peer Rejection, Pra = Counselor Praised Behavior. TPQ = Treatment Participation Questionnaire, ARCQ = Adolescent Relapse Coping Questionnaire, SW = social worker. CD = conduct disorder symptom count, AL = Alcohol Ladder, ML = Marijuana Ladder, D = depressive symptoms, P = positive scale, N = negative scale, SE = self-efficacy, PS = problem-solving, SC = self-critical thinking, AB = abstinence-focused coping. X = non-significant.
Point-biserial correlation between continuous and dichotomous variables.
Significant after Bonferroni adjustment for (14×15) comparisons (.05/210).
TABLE 4.
Correlations of Group Process at Session 3 with Relevant Constructs at Session 10.
| TPQ |
ARC-Q |
BSCQ |
||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A | C | SW | ||||||||||
| Dev | P | N | P | N | P | N | SE | PS | SC | AB | ALC | MJ |
| A | X | .28 | X | .21 | X | X | X | X | X | X | −.24 | −.16 |
| p≤ | .001 | .011 | .004 | .034 | ||||||||
| C | X | .30* | −.38* | .64* | X | .15 | X | X | −.18 | X | −.16 | −.23 |
| p≤ | .001 | .001 | .001 | .049 | .020 | .033 | .005 | |||||
| O | X | .19 | −.24 | .36* | X | X | X | X | X | X | X | X |
| p≤ | .002 | .005 | .001 | |||||||||
| Pos | ||||||||||||
| A | .27 | −.16 | X | X | X | X | X | X | X | X | .20 | X |
| p≤ | .001 | .041 | .012 | |||||||||
| C | .21 | X | .34* | −.19 | .27 | −.19 | X | X | X | X | X | .18 |
| p≤ | .010 | .001 | .018 | .001 | .017 | .023 | ||||||
| O | .21 | x | X | X | X | X | X | X | X | X | X | X |
| p≤ | .012 | |||||||||||
| Rej | ||||||||||||
| A | X | .18 | −.22 | .24 | X | X | X | X | X | X | X | X |
| p≤ | .023 | .006 | .004 | |||||||||
| C | X | X | X | .27 | X | X | −.15 | X | X | X | X | X |
| p≤ | .001 | .045 | ||||||||||
| Oa | X | X | −.18 | .16 | X | X | X | X | X | X | X | −.20 |
| p≤ | .029 | .040 | .015 | |||||||||
| Pra | ||||||||||||
| A | .51* | −.16 | .17 | X | .26 | X | X | .32* | .28 | .31* | .19 | X |
| p≤ | .001 | .039 | .028 | .002 | .001 | .001 | .001 | .017 | ||||
| C | X | X | .18 | X | .26 | X | X | X | X | X | X | X |
| p≤ | .022 | .002 | ||||||||||
| O | .18 | X | X | X | X | X | X | X | .15 | X | X | X |
| p≤ | .027 | .048 | ||||||||||
| Cona | ||||||||||||
| A | X | X | X | X | X | X | .19 | X | X | X | X | X |
| p≤ | .016 | |||||||||||
| C | X | .19 | X | X | X | X | X | X | X | X | X | X |
| p≤ | .017 | |||||||||||
| O | .39* | X | −.17 | .24 | X | X | X | .33* | .19 | .31* | X | X |
| p≤ | .001 | .035 | .004 | .001 | .020 | .001 | ||||||
A = adolescent, C = counselor, O = observer. Dev = Deviancy, Con = Counselor Connection, Pos = Positive Behavior, Rej = Peer Rejection, Pra = Counselor Praised Behavior. TPQ = Treatment Participation Questionnaire, ARCQ = Adolescent Relapse Coping Questionnaire, SW = social worker, BSCQ = Brief Situational Confidence Questionnaire. P = positive scale, N = negative scale, SE = self-efficacy, PS = problem-solving, SC = self-critical thinking, AB = abstinence-focused coping, ALC = alcohol, MJ = marijuana. X = non-significant.
Point-biserial correlation between continuous and dichotomous variables.
Significant after Bonferroni adjustment for (12x15) comparisons (.05/180).
Inter-Rater Agreement
For observers, to calculate Intra-class Correlation Coefficients (ICCs) a 2-way random model was chosen because we wanted results to generalize to other samples and raters; α was set at .01 (.05/5, for 5 analyses). Type=consistency was chosen because it was felt that consistency between raters was an adequate bar to set (as compared to absolute agreement of ratings). Average measures was chosen because we are combining across 3 blocks within a session. ICCs for the scales at session 3 were as follows: Dev3=.74, p≤ .001; Pos3=.82, p≤ .001; Rej3=.63, p≤ .007; Pra3=.65, p≤ .005; and Con3=.85, p≤ .001. These ICCs indicate adequate or better agreement.26,27
Stability Over Time
For adolescents, test-retest stabilities between sessions 3 and 10 were calculated with □ set at .01 (.05/5): r(Dev-A)=.42, p≤ .001; r(Pos-A)=.39, p≤ .001; r(Rej-A)=.13, p≥ .155; r(Pra-A)=.44, p≤ .001;and □(Con-A)= −.020, p≥ .825 (ϕ provided for dichotomous data; 96% felt connected at both sessions). Similarly for counselors:r(Dev-C)=.62, p≤ .001; r(Pos-C)=.46, p≤ .001; r(Rej-C)=.41, p≤ .001; r(Pra-C)=.51, p≤ .001; and ϕ(Con-C)=.93, p≤ .001. To assess stability between 3rd and 10th session, ICC’s were calculated for observers, again with α =.01 (.05/5). ICCs were obtained (instead of Pearson’s r) for the observer version, since different raters were sometimes used at the 2 assessment points. A 1-way model, type=consistency, average measures was chosen because observers may have varied between both sessions, we did not expect absolute agreement over time, and measures were obtained at two points. The ICCs were: Dev-O=.28, p≤ .070; Pos-O=.42, p≤ .008; Rej-O=.44, p≤ .005; Pra-O=.38, p≤ .017; and Con-O=.22, p≥ .128. These ICCs indicate adequate or better agreement26,27 except for two of the adolescent scales and one of the observer scales.
Treatment by Time Analysis
Treatment by time multivariate analysis of covariance (MANCOVA) was conducted on GP-IL scales. No predictions were made with respect to treatment, time or interaction effects; however, it was expected that results would evidence a discernable pattern (for example, reduction in deviancy across all raters over time, consistent difference in rejection between treatments among raters, etc.). Covariates included age, conduct disorder symptom count and gender (since treatments were segregated by gender).
For adolescents and counselors, only the four continuous scales (Dev, Pos, Rej, Pra) were analyzed as dependent variables, so that using the Bonferroni correction, α = .013 (.05/4). For observers, three continuous scales were available for analyses (Dev, Pos, Pra), leaving α = .017 (.05/3). To conserve space, only significant results are presented; see Table 5 for summary statistics. Significant treatment effects were obtained for the adolescent Rej scale: F(1, 121) = 5.41, p≤ .022, partial η2 = .043 (small-medium effect size [ES]26). For the counselor Rej scale, significant treatment effects were obtained: F(1, 124) = 18.45, p≤ .001, partial η2 = .130 (largeES). For the counselor Dev scale, treatment effects were found withF(1, 124) = 13.52, p≤ .001, partial η2 = .098 (medium-large ES); and effects for time indicatedF(1, 125) = 4.43, p≤ .037, partial η2 = .034 (small-medium ES). Finally, significant treatment effects were obtained for the observerDev scale: F(1, 77) = 11.06, p≤ .001, partial η2 = .126 (large ES). No significant effects were obtained for the treatment by time interaction. WithBonferroni correction, significant treatment effects remained only forcounselor Dev, counselor Rej, and observer Dev scales.
TABLE 5.
Descriptive Statistics on Scales Producing Significant Multivariate Analyses of Covariance Results
| 3rd
Assessment |
10th
Assessment |
||||
|---|---|---|---|---|---|
| M | SD | M | SD | ||
| Rej-A | |||||
| MI/CBT (n = 61) | .20 | .40 | .21 | .36 | |
| RT/SET (n = 64) | .14 | .35 | .06 | .18 | |
| Rej-C | |||||
| MI/CBT (n = 62) | .45 | .52 | .60 | .62 | |
| RT/SET (n = 66) | .22 | .31 | .30 | .35 | |
| Dev-C | |||||
| MI/CBT (n = 62) | 1.29 | .76 | 1.62 | .84 | |
| RT/SET (n = 66) | .98 | .79 | 1.04 | .80 | |
| Dev-O | |||||
| MI/CBT (n = 42) | .43 | .54 | .43 | .39 | |
| RT/SET (n = 39) | .17 | .24 | .21 | .39 | |
M = mean, SD = standard deviation, n=number of participants, Rej = Rejection, Dev = Deviancy, A = adolescent, C = counselor, O = observer, MI/CBT = Motivational Interviewing and Cognitive Behavior Therapy, RT/SET = Relaxation Therapy and Substance Education with Twelve-Step Programming.
To analyze dichotomous scales (observer-rated peer rejection; adolescent-, counselor- and observer-rated connectedness), logistic regressions were employed. For adolescents and counselors no □ corrections were used. For observers, since two analyses were run, the Bonferroni correction yielded α = .025 (.05/2). Results indicate no effects for treatment, time or treatment by time.
Incremental Validity
Incremental validity was determined by employing hierarchical regression as presented in Table 6. The dependent variables (DVs), in separate analyses, included TPQ positive and negative scales for adolescent, counselor and social worker at the 3rd session assessment. Step 1 included the following baseline covariates: Age, conduct disorder symptom count, Ladder-Alcohol (since motivation to change relates to outcome),16 and baseline version of the DV. Step 2 included the GP-IL variables at the 3rd session assessment. Step 2 did not include the Con index for A, C and O. While other constructs on the GP-IL questionnaire are assessed via a series of Likert questions, the Con index is a simple question asking to which counselor the adolescent is connected. Because of this difference in manner of assessment, the Con indicator did not fit conceptually with the other measures being entered on step 2.
TABLE 6.
Incremental Validity with Treatment Participation as the Dependent Variable
| Standard Betas |
|||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Step | R2 | RA2 | Rch2 | F | df | <p | Dev (<p) |
Pos (<p) |
Rej (<p) |
Pra (<p) |
|
| 3rd Session GP-IL Related to 3rd Session Constructs | |||||||||||
| TPQA-N3a | |||||||||||
| 1) TPQA-NBL, age, AL, CD | .11 | .08 | .11 | 3.81 | 4,129 | .006 | na | na | na | na | |
| 2) GP-ILA-3 | .20 | .15 | .09 |
(Fch) |
3.90 3.67 |
8,125 4,125 |
.001 .007* |
.27 (.003) |
NS | NS | NS |
| 1) TPQA-NBL age, AL, CD | .11 | .08 | .11 | 3.81 | 4,129 | .006 | na | na | na | na | |
| 2) GP-ILC-3 | .24 | .19 | .13 |
(Fch) |
4.83 5.33 |
8,125 4,125 |
.001 .001* |
.37 (.001) |
NS | NS | NS |
| TPQA-P3 a | |||||||||||
| 1) TPQA-NBL age, AL, CD | .33 | .31 | .33 | 15.98 | 4,129 | .001 | na | na | na | na | |
| 2) GP-ILA-3 | .46 | .43 | .13 |
(Fch) |
13.52 7.73 |
8,125 4,125 |
.001 .001* |
NS | NS | NS | .32 (.001) |
| 1) TPQA-PBL, age, AL, CD | .33 | .31 | .33 | 15.98 | 4,129 | .001 | na | na | na | na | |
| 2) GP-ILC-3 | .40 | .36 | .07 |
(Fch) |
10.33 3.46 |
8,125 4,125 |
.001 .010* |
−.18 (.022) |
.20 (.011) |
NS | NS |
| TPQC-N3a | |||||||||||
| 1) TPQA-NBL, age, AL, CD | .05 | .02 | .05 | 1.56 | 4,133 | .189 | na | na | na | na | |
| 2) GP-ILC-3 | .72 | .70 | .67 |
(Fch) |
40.43 75.79 |
8,129 4,129 |
.001 .001* |
.81 (.001) |
−.26 (.001) |
NS | .11 (.040) |
| 1) TPQA-NBL, age, AL, CD | .05 | .01 | .05 | 1.41 | 4,120 | .236 | na | na | na | na | |
| 2) GP-ILO-3 | .18 | .12 | .13 |
(Fch) |
3.11 4.64 |
8,116 4,116 |
.003 .002* |
.33 (.001) |
NS | NS | NS |
| TPQC-P3a | |||||||||||
| 1) TPQA-PBL, age, AL, CD | .06 | .03 | .06 | 2.22 | 4,133 | .070 | na | na | na | na | |
| 2) GP-ILC-3 | .54 | .51 | .48 |
(Fch) |
19.07 33.72 |
8,129 4,129 |
.001 .001* |
−.27 (.001) |
.49 (.001) |
.20 (.003) |
.28 (.001) |
| TPQS-N3a | |||||||||||
| 1) TPQA-NBL, age, AL, CD | .04 | .01 | .04 | 1.48 | 4,130 | .211 | na | na | na | na | |
| 2) GP-ILC-3 | .12 | .06 | .07 |
(Fch) |
2.08 2.61 |
8,126 4,126 |
.042 .039 |
.20 (.035) |
−.20 (.034) |
NS | NS |
| TPQS-P3a | |||||||||||
| 1) TPQA-PBL, age, AL, CD | .04 | .01 | .04 | 1.33 | 4,130 | .263 | na | na | na | na | |
| 2) GP-ILC-3 | .13 | .07 | .09 |
(Fch) |
2.28 3.14 |
8,126 4,126 |
.026 .017* |
NS | .18 (.052) |
NS | .17 (.058) |
| 3rd Session GP-IL Related to 10th Session Constructs | |||||||||||
| TPQA-N10a | |||||||||||
| 1) TPQA-NBL, age, AL, CD | .16 | .13 | .16 | 5.50 | 4,120 | .001 | na | na | na | na | |
| 2) GP-ILA-3 | .28 | .23 | .12 |
(Fch) |
5.54 4.88 |
8,116 4,116 |
.001 .001* |
.21 (.017) |
NS | .20 (.019) |
NS |
| 1) TPQA-NBL, age, AL, CD | .16 | .13 | .16 | 5.63 | 4,123 | .001 | na | na | na | na | |
| 2) GP-ILC-3 | .23 | .18 | .07 |
(Fch) |
4.42 2.87 |
8,119 1,119 |
.001 .026 |
.23 (.013) |
NS | NS | NS |
| TPQA-P10a | |||||||||||
| 1) TPQA-PBL, age, AL, CD | .25 | .22 | .25 | 9.82 | 4,120 | .001 | na | na | na | na | |
| 2) GP-ILA-3 | .42 | .38 | .17 | 10.49 8.65 |
8,116 4,116 |
.001 .001* |
NS | .39 (.001) |
NS | NS | |
| TPQC-N10a | |||||||||||
| 1) TPQA-NBL, age, AL, CD | .07 | .04 | .07 | 2.20 | 4,120 | .073 | na | na | na | na | |
| 2) GP-ILA-3 | .15 | .09 | .08 |
(Fch) |
2.45 2.59 |
8,116 4,116 |
.017 .040 |
NS | NS | .23 (.011) |
NS |
| 1) TPQA-NBL, age, AL, CD | .07 | .04 | .07 | 2.26 | 4,123 | .067 | na | na | na | na | |
| 2) GP-ILC-3 | .47 | .44 | .40 |
(Fch) |
13.23 22.62 |
8,119 4,119 |
.001 .001* |
.59 (.001) |
−.18 (.014) |
NS | NS |
| 1) TPQA-NBL, age, AL, CD | .07 | .04 | .07 | 2.07 | 4,113 | .089 | na | na | na | na | |
| 2) GP-ILO-3 | .19 | .13 | .12 |
(Fch) |
3.17 4.05 |
8,109 4,109 |
.003 .001* |
.32 (.001) |
NS | NS | NS |
| TPQC-P10a | |||||||||||
| 1) TPQA-PBL, age, AL, CD | .02 | .00 | .02 | .68 | 4,123 | .610 | na | na | na | na | |
| 2) GP-ILC-3 | .27 | .22 | .25 |
(Fch) |
5.50 10.13 |
8,119 4,119 |
.001 .001* |
−.38 (.001) |
.28 (.014) |
NS | NS |
| 1) TPQA-PBL, age, AL, CD | .02 | .00 | .02 | .62 | 4,113 | .648 | na | na | na | na | |
| 2) GP-ILO-3 | .12 | .05 | .10 |
(Fch) |
1.83 3.00 |
8,109 4,109 |
.079 .022 |
−.23 (.016) |
NS | NS | NS |
| TPQS-P10a | |||||||||||
| 1) TPQA-PBL, age, AL, CD | .02 | .00 | .02 | .50 | 4,116 | .733 | na | na | na | na | |
| 2) GP-ILA-3 | .10 | .04 | .09 |
(Fch) |
1.62 2.70 |
8,112 4,112 |
.128 .034 |
NS | NS | NS | .26 (.011) |
| 1) TPQA-PBL, age, AL, CD | .02 | .00 | .02 | .52 | 4,119 | .723 | na | na | na | na | |
| 2) GP-ILC-3 | .13 | .07 | .12 |
(Fch) |
2.23 3.89 |
8,115 4,115 |
.30 .005* |
NS | .20 (.041) |
NS | NS |
TPQA, TPQC, TPQS = Treatment Participation Questionnaire-adolescent, counselor and social worker versions, respectively.
N, P = negative and positive components, respectively.
BL, 3 and 10 = baseline, 3rd and 10th session assessments, respectively.
AL, CD = Alcohol Ladder and conduct disorder symptom count, respectively.
GP-IL A-3, C-3, O-3 = Group Process-Individual Level questionnaire-adolescent, counselor and observer versions, respectively, at the 3rd session assessment.
GP-IL contains Deviancy (Dev), Positive (Pos), Reject (Rej), and Praise (Pra) indicators.
Fch = change in F-test
Dependent variable
na = not applicable
NS = non-significant
Significant after employing, separately for concurrent and predictive validity, the Bonferroni correction for 3 regressions per DV (α = .017; .05/3)
For each DV, analyses were run separately for the adolescent, counselor and observer versions of the GP-IL measure. For example, to predict adolescent ratings of negative treatment participation at the 3rd session, step 1 included adolescent baseline negative treatment participation, age, Alcohol Ladder rating, and conduct disorder symptom count, which provided for R2 = .11, F(4,129)=3.81, p≤ .006. Step 2 included adolescent ratings of deviancy, positive behavior, peer rejection and counselor praise, yielding R2 = .20, F(8,125)=3.90, p≤.001. This step provided Fch(4,125) = 3.67, p≤ .007, with standardized βDev = .27, p≤ .003. Then, similar analyses were run, but with counselor group process variables entered on step 2 which provided: R2 = .24, F(8,125) = 4.83, p≤.001, Fch(4,125) = 5.33, p≤ .001, and βDev = .37, p≤ .001. Analyses entering observer group process variables on step 2 were not significant. Counselors and social workers did not complete treatment participation ratings on adolescents at baseline. Therefore, when DVs involved counselor or social worker participation ratings, the adolescent version at baseline was entered as the covariate on step 1. Because 3 hierarchical regressions were run for each DV, a Bonferroni correction of α =.017 (.05/3) was applied. Prior to the correction, Table 6 shows 9 significant regressions for analyses of incremental concurrent validity, whereas after the correction, 8 remain significant. Incremental predictive validity analyses relating GP-IL variables to constructs as measured at a future assessment point were performed in the same manner and are also in Table 6. For these analyses treatment participation at the 10th session assessment was the DV. Analyses were repeated, entering interest in changing marijuana use (Ladder-M) on step 1. Results are consistent with those presented in Table 7 and are not presented to conserve space.
DISCUSSION
The three versions of the GP-IL for the most part show good psychometric properties, but with some exceptions worth noting. Internal consistencies for scales ranged from adequate to excellent (.56-.96), with the Counselor Praise scale having less acceptable internal consistencies (.45-.69). Correlations among scales evidenced good convergent validity, which supports the construct validity of the GP-IL measure. When correlating scales to other constructs assessed at the same time, observer measures did not perform as well as counselor and adolescent measures, and similarly little evidence was found to support the convergent validity of the Peer Rejection indicator. When correlating group process scales during the 3rd session assessment with constructs assessed following the 10th session, convergent validity was found for the adolescent, counselor and observer versions of Deviancy, Positive Behavior, Counselor Praise and Counselor Connectedness scales, but not for the Peer Rejection index. Inter-rater reliabilities for observers were adequate (.63-.85), and scales were fairly stable from sessions 3 to 10 (.39-.93), except for adolescent reports on peer rejection and connectedness, and observer reports of deviancy, praise, and connectedness.
MANCOVAs indicate that both Deviancy and Rejection scales are sensitive to type of treatment group with less rejection and deviancy occurring in a group designed to be less interactive and more didactic. Because counselors conducted both intervention types and close supervision was used, it is unlikely that results are due to counselor effects or poor treatment implementation. Regressions indicate that adolescent group process ratings offer some limited incremental validity as concurrent and prospective predictors of treatment participation with significant associations for deviancy and praise, and relatively fewer associations for peer rejection and positive behavior. Counselor group process ratings provide the strongest association with treatment engagement, with multiple significant associations found for ratings of deviancy, positive involvement and counselor praise in particular. In contrast, there was essentially no significant evidence for the incremental validity of observer group process ratings, with the exception of observer deviancy ratings. Thus with respect to incremental validity analyses, it appears that counselor group process ratings show the strongest support, and that ratings of peer rejection show little support across respondents whereas deviancy ratings show the most support across respondents.
The above analyses of the GP-IL scales indicate that observer, counselor and adolescent scales generally have sound psychometric properties, but that psychometrics are best for counselor scales. Because adolescent and counselor versions are less labor intensive and have better psychometric properties, they appear to be preferable to the observer version. In addition, while the counselor version has the best validity evidence, it is likely more efficient and practical to have adolescents complete a form than to ask counselors to complete a form on each group participant. Therefore, researchers or practitioners choosing among forms may need to weigh the validity evidence against available resources.
It is unclear why the counselor version is best. It is possible that observers miss subtle interactions by not being present, and that youth are not good at observing behaviors in themselves (perhaps lacking introspection); whereas counselors are present and can notice much that is missed by youth and observers. Replication is needed to see if the counselor version is best in other samples. Future studies might improve the item content of some scales and develop cut-scores to determine, for example, maximum tolerable levels of deviancy or minimal levels of positive involvement. The Bonferroni correction was used to control for family-wise error (FWE), therefore, some significant relationships among constructs may have been ignored. On the other hand, even with this conservative approach, the instrument was found to have sound psychometric properties, bolstering confidence in its usefulness. Future studies may use less conservative methods, such as the Benjamini & Hochberg procedure28, to control for (FWE).
This study is important for several reasons. It provides intensive psychometric analyses of an instrument that examines group processes as measured at the level of the individual. Although the original version was for use by observers, we created counselor and adolescent versions to access additional vantage points and found that these had better psychometric properties. Thus, adolescent or counselor ratings can be used in place of observer ratings to save costs. Tracking group processes may be important to researchers and clinicians during intervention, and these measures offer a method of doing so efficiently. Adolescents are frequently aggregated in groups, especially incarcerated substance abusers; therefore, being able to track group processes is important. Using a measure with demonstrated reliability and validity can assist in this endeavor.
Acknowledgments
The authors would like to thank Suzanne Sales and Rick Palumbo for their tireless work on this project. We would also like to thank Dr. Tom Dishion for his interest and support.
FUNDING
National Institute on Drug Abuse - Grant Number: R01 DA-018851, PI-Stein - RFP. The funding agency was not involved in the work reported in this manuscript or in the composition of the submission.
REFERENCES
- 1.Teplin LA, Abram KM, McClelland GM, Dulcan MK, Mericle AA. Psychiatric disorders in youth in juvenile detention. Arch Gen Psychiatry. 2002;59:1133–1143. doi: 10.1001/archpsyc.59.12.1133. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Mulvey EP, Schubert CA, Chassin L. Juvenile Justice Bulletin. U.S. Department of Justice; Washington, D.C.: 2010. Substance use and delinquent behavior among serious adolescent offenders. [Google Scholar]
- 3.The Correctional Association of New York . Treatment behind bars: Substance abuse treatment in New York prisons. The Correctional Association of New York; New York, NY: 2011. [Google Scholar]
- 4.Dishion TJ, McCord J, Poulin F. When interventions harm: Peer groups and problem behavior. Am Psychol. 1999;54(9):755–764. doi: 10.1037//0003-066x.54.9.755. [DOI] [PubMed] [Google Scholar]
- 5.Dishion TJ, Poulin F, Burraston B. Peer group dynamics associated with iatrogenic effects in group interventions with high-risk young adolescents. In: Nagle DW, Endley CA, editors. The role of friendship in psychological adjustment. Jossey-Bass; San Francisco: 2001. pp. 79–92. [DOI] [PubMed] [Google Scholar]
- 6.Mager W, Milich R, Harris MJ, Howard A. Intervention groups for adolescents with conduct problems: Is aggregation harmful or helpful? J Abnorm Child Psychol. 2005;33(3):349–362. doi: 10.1007/s10802-005-3572-6. [DOI] [PubMed] [Google Scholar]
- 7.Poulin F, Dishion TJ, Burraston B. 3-year iatrogenic effects associated with aggregating high-risk adolescents in cognitive-behavioral preventative interventions. Appl Dev Sci. 2001;5(4):214–224. [Google Scholar]
- 8.Waldron HB, Kaminer Y. On the learning curve: The emerging evidence supporting cognitive-behavioral therapies for adolescent substance abuse. Addiction. 2004;99(suppl 2):93–105. doi: 10.1111/j.1360-0443.2004.00857.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.French MT, Zavala SK, McCollister KE, Waldron HB, Turner CW, Ozechowski TJ. Cost-effectiveness analysis of four interventions for adolescents with a substance use disorder. J Subst Abuse Treat. 2008;34(3):272–281. doi: 10.1016/j.jsat.2007.04.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Sampl S, Kadden R. Motivational Enhancement Therapy and CognitiveBehavioral Therapy for Adolescent Cannabis Users: 5 Sessions, CannabisYouth Treatment (CYT) Series, Volume 1. Center for Substance Abuse Treatment, Substance Abuse and Mental Health ServicesAdministration; Rockville, MD: 2001. DHHS Pub. No. (SMA) 01–3486. [Google Scholar]
- 11.Webb C, Scudder M, Kaminer Y, Kadden R. The Motivational Enhancement Therapy and Cognitive Behavioral Therapy Supplement: 7 Sessions of Cognitive Behavioral Therapy for Adolescent Cannabis Users, Cannabis Youth Treatment (CYT) Series, Volume 2. Center for Substance Abuse Treatment, SubstanceAbuse and Mental Health Services Administration; Rockville, MD: 2002. DHHSPub. No. (SMA) 02-3659. [Google Scholar]
- 12.Stein LAR. Cognitive behavior therapy for alcohol and marijuana use: A manual for treating incarcerated teens. 2005 Unpublished manual, Brown University, Providence, RI; NIDA/NIAAA, Rockville, MD. [Google Scholar]
- 13.Rose M, Klein J, Stein LAR, Lebeau-Craven R, Justus A. Substance education treatment: A manual for treating incarcerated adolescents. 2005 Unpublished manual, Brown University, Providence, RI; NIDA/NIAAA, Rockville, MD. [Google Scholar]
- 14.Kessler R, Andrews G, Mroczek D, Ustun B, Wittchen H. The World Health Organization Composite International Diagnostic Interview Short Form (CIDI-SF) Int J Methods Psychiatr Res. 1997;7:171–185. [Google Scholar]
- 15.Slavet J, Stein LAR, Colby S, et al. The Marijuana Ladder: Measuring motivation to change marijuana use in incarcerated adolescents. Drug Alcohol Depend. 2006;83:42–48. doi: 10.1016/j.drugalcdep.2005.10.007. PMCID: PMC2754131. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Clair M, Stein LAR, Martin R, et al. Motivation to change alcohol use and treatment engagement in incarcerated youth. Addict Behav. 2011;36(6):674–680. doi: 10.1016/j.addbeh.2011.01.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Breslin FC, Sobell LC, Sobell MB, Buchan G. A comparison of a brief and long format for the Situational Confidence Questionnaire; 31st Annual Meeting of the Association for the Advancement of Behavior Therapy; Miami Beach, FL. Nov, 1997. [Google Scholar]
- 18.Radloff LS. The use of the Center for Epidemiologic Studies Depression Scale in adolescents and young adults. J Youth Adolesc. 1991;20:149–166. doi: 10.1007/BF01537606. [DOI] [PubMed] [Google Scholar]
- 19.Stein LAR, Colby SM, Barnett NP, et al. Enhancing substance abuse treatment engagement in incarcerated adolescents. Psychol Serv. 2006;3:25–34. doi: 10.1037/1541-1559.3.1.0. PMCID: PMC2898284. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Myers M, Brown S. Coping responses and relapse among adolescent substance abusers. J Subst Abuse. 1990;2:177–189. doi: 10.1016/s0899-3289(05)80054-9. [DOI] [PubMed] [Google Scholar]
- 21.Myers MG, Brown SA, Mott MA. Coping as a predictor of adolescent substance abuser treatment outcome. J Subst Abuse. 1993;5:15–29. doi: 10.1016/0899-3289(93)90120-z. [DOI] [PubMed] [Google Scholar]
- 22.Myers MG, Brown SA, Mott MA. Preadolescent conduct disorder behaviors predict relapse and progression of addiction for adolescent alcohol and drug abusers. Alcohol ClinExp Res. 1995;19(6):1528–1536. doi: 10.1111/j.1530-0277.1995.tb01019.x. [DOI] [PubMed] [Google Scholar]
- 23.Reavy R, Stein LAR, Paiva A, et al. Validation of the delinquent activities scale for incarcerated adolescents. Addict Behav. 2012;37:875–879. doi: 10.1016/j.addbeh.2012.03.007. PMCID:PMC3356453. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Reavy R, Stein LAR, Quina K, et al. Assessing conduct disorder: A new measurement approach. J Correct Health Care. 2014;20(1):4–17. doi: 10.1177/1078345813505448. DOI: 10.1177/1078345813505448. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Tabachnick B, Fidell L. Using MultivariateStatistics. 3rd HarperCollins; New York: 1996. [Google Scholar]
- 26.Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd Lawrence Erlbaum Associates; Hillsdale, NJ: 1988. [Google Scholar]
- 27.Barnett J. Effect Size and Measures of Association. University of Alabama; Birmingham: 2006. Summer Evaluation Institute sponsored by The American Evaluation Assoc. and Centers for Disease Control and Prevention. [Google Scholar]
- 28.Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J R Stat Soc. 1995;57(1):289–300. [Google Scholar]