Effectiveness of a pragmatic school-based universal resilience intervention in reducing tobacco, alcohol and illicit substance use in a population of adolescents: cluster-randomised controlled trial

Abstract

Objectives

Initiation of tobacco, alcohol and illicit substance use typically occurs during adolescence, with the school setting recommended to reduce adolescent substance use. Strengthening individual (eg, problem solving) and environmental (eg, caring relationships at school) resilience protective factors of adolescents has been suggested as a strategy for reducing substance use by adolescents; however, few studies have examined this potential. A study was conducted to investigate the effectiveness of a pragmatic school-based universal ‘resilience’ intervention in reducing the prevalence of tobacco, alcohol and illicit substance use, and increasing the individual and environmental protective factors of students.

Design

A cluster-randomised controlled trial.

Setting

Thirty-two Australian secondary schools (20 intervention; 12 control).

Participants

Cohort of grade 7 students followed-up in grade 10 (2014; aged 15–16 years).

Intervention

A pragmatic intervention involving school staff selection and implementation of available programmes and resources targeting individual and environmental ‘resilience’ protective factors for all grade 7–10 students was implemented in schools (2012–2014). School staff were provided implementation support.

Measurements

An online survey collected baseline and follow-up data for primary outcomes: tobacco (ever, recent) and alcohol (ever, recent, ‘risk’) use, and secondary outcomes: marijuana and other illicit substance use, and individual (six-factor subscales, aggregate) and environmental (three-factor subscales, aggregate) protective factor scores. Generalised and linear mixed models examined follow-up differences between groups.

Results

Follow-up data from 2105 students (intervention=1261; control=844; 69% of baseline cohort) were analysed. No significant differences were found between intervention and control students for any primary (ever tobacco: OR 1.25, 95% CI 0.92 to 1.68, p=0.14; recent tobacco: OR 1.39, 95% CI 0.84 to 2.31, p=0.19; recent ever alcohol: OR 1.11, 95% CI 0.83 to 1.48, p=0.46; alcohol: OR 1.13, 95% CI 0.78 to 1.62, p=0.51; ‘risk’ alcohol: OR 0.98, 95% CI 0.70 to 1.36, p=0.89) or secondary outcomes (marijuana: OR 1.12, 95% CI 0.74 to 1.68, p=0.57; other illicit substance: OR 1.19, 95% CI 0.67 to 2.10, p=0.54; individual protective factors: MD=0, 95% CI −0.07 to 0.06, p=0.89; environmental protective factors: MD: −0.02, 95% CI −0.09 to 0.06, p=0.65).

Conclusions

The universally implemented pragmatic school-based intervention was not effective in reducing the prevalence of tobacco, alcohol or illicit substance use, or in increasing the protective factors of students.

Trial registration

Australia and New Zealand Clinical Trials Register reference: ACTRN12611000606987

Strengths and limitations of this study.

• This study represents a comprehensive examination using the gold standard study design for school-based studies to examine the potential of a universal school-based resilience protective factor intervention in reducing the tobacco, alcohol and illicit substance use of adolescents.

• Major strengths of this study include: the cluster-randomised controlled study design, the large sample size of participating students, the collection of individual outcome data as well as process data to assess intervention implementation and the use of statistical methods that both accounted for the clustering of student outcome data and sensitivity analyses of data via intention-to-treat principles including multiple imputation to account for missing data.

• Although the study found a high rate of student attrition (31%), such attrition is typical for school-based research, did not differ between treatment groups and had little impact on the estimated power of the study (difference of 0.3%–0.4%).

• The study was reliant on adolescent self-report of substance use and subject to the known limitations of self-report in this population. While the planned validation of tobacco use by adolescents was not supported by schools, strategies were implemented to increase the validity of adolescent report including a web-based survey and confidential participation by students.

Background

Tobacco, alcohol and illicit substance use are responsible for 9% of the global disease burden,¹ 12% of deaths worldwide² and significant health and societal costs.^3–6 Initiation of tobacco, alcohol and illicit substance use in high-income countries generally occurs during adolescence,^7–9 with earlier use associated with greater dependence in adulthood.¹ While data from the USA and Australia show a declining trend in adolescent substance use,^{9 10} a considerable proportion of adolescents (aged 11–17 years) continue to report such use; 23%–45% having smoked a cigarette, 43%–74% having consumed an alcoholic drink and 7%–40% having used an illicit substance.^9–11

Schools represent an opportune setting for interventions to prevent adolescent substance use as they provide access to large numbers of adolescents for prolonged periods, and have curricula and policies that seek to promote student health and well-being.^{12 13} As a consequence, substance use prevention interventions delivered to all students in a school or classroom regardless of risk (that is universal)^{14 15} are common and supported by governments worldwide to reduce the prevalence of adolescent substance use.^16–19 Despite policies recommending comprehensive approaches to substance use prevention address protective factors of substance use^{17 19–21} and ‘resilience’,^{17 19} such policies do not provide guidance regarding the specific factors or resilience strategies that should be targeted or the manner in which they should be addressed. Possibly as a result, it is reported that schools frequently develop their own programmes,²² do not implement evidence-based programmes or implement existing evidence-based programmes²³ and make significant adaptations to cater for local contexts.²⁴ The extent to which such an approach can realise its intended benefits has not been reported.

Evidence from cross-sectional studies suggests a range of individual factors including self-efficacy, problem solving, communication and self-awareness are protective of adolescent substance use; as has evidence regarding environmental factors such as caring relationships with adults and peers, and meaningful participation in home, school and community settings.^25–37 Such factors have similarly been found to be protective of a person’s ‘resilience’,^38–40 most broadly defined as the process of, capacity for or outcome of successful adaptation in the context of risk or adversity.^40–42

Various randomised controlled trials have assessed the effectiveness of resilience protective factor interventions on substance use.⁴³ These have primarily addressed either resilience protective factors as a component of a broader intervention approach,^44–60 combined universal and targeted interventions,^{61 62} combined parent and school-based strategies⁶³ or involved elementary school aged students only.⁶⁴ However, only one controlled trial that assessed the effectiveness of a universal school-based intervention focused solely on the enhancement of both individual and environmental resilience protective factors in reducing the prevalence of adolescent or secondary school-aged students substance use. The cluster-randomised controlled trial conducted in 26 Australian secondary schools, investigated the effectiveness of a 3-year whole-of-school intervention delivered by schools (ie, pragmatic) targeting a number of individual and environmental protective factors in preventing tobacco, alcohol and marijuana use in a cohort of students.⁴⁶ Outcomes were assessed at baseline, midintervention (after 1 year of intervention) and following intervention completion. Despite promising midintervention results for tobacco use, at follow-up the CIs for the adjusted ORs for tobacco, alcohol or marijuana use outcomes indicated a non-significant result.⁴⁵

Given the limited evidence regarding the effectiveness of universal interventions promoting protective factors as a means of reducing adolescent student substance use, a cluster randomised controlled trial was conducted to determine the effectiveness of a secondary school staff-delivered pragmatic intervention targeting such protective factors in reducing the prevalence of tobacco and alcohol use (primary outcomes) and marijuana and illicit substance use, and in increasing individual and environmental protective factors (secondary outcomes).

Methods

Study design and setting

A cluster randomised controlled trial was conducted in secondary schools in one health district of New South Wales, Australia. Outcome assessments were conducted with a cohort of students at baseline (when students were in grade 7, aged 12–13 years) and at follow-up (when students were in grade 10). Approximately 1 14 000 people aged 10–19 years reside in metropolitan, regional, rural and remote areas within the district.^{65 66} Relevant ethics committee approvals were obtained (Hunter New England Health Ref: 9/11/18/4.01; University of Newcastle Ref: H-2010–0029). Further study details and assessment of other registered outcomes are reported elsewhere.^{67 68}

Participants and recruitment

Schools

A national schools database⁶⁹ identified 172 schools with secondary enrolments within the study area. Schools were eligible if they: were a Government or Catholic secondary school located within a socioeconomically disadvantaged local government area,⁷⁰ had enrolments in grades 7–10 (aged 12–16 years) and had more than 400 total student enrolments. Schools were ineligible if they were: single gender, independent (private), special needs, selective, central (for students aged 5–18 years) or boarding schools.

Randomisation of schools

Eligible schools were approached in random order until a quota of 32 schools consented. Consenting schools were stratified according to participation in a government disadvantaged schools initiative (yes/no)⁷¹ and school size (medium 400–800/large >800), then randomly allocated to intervention or control in a 20:12 block design ratio by an independent statistician using a random number function in Microsoft Excel prior to baseline data collection (the number of intervention schools were increased from planned 12 to 20 following stakeholder consultation).

Students

All students enrolled in grade 7 (first year at secondary school) were eligible to participate in data collection and active parental consent for student participation was sought via a mailed study information pack. A free call number was provided for parents who wished to decline. After 2 weeks, non-responding parents were prompted via telephone by school-affiliated staff who were blind to group allocation.

School staff

Selected school staff (deputy principal, head teachers for student welfare and five key subject areas and the Aboriginal Education Coordinator or other Aboriginal staff member) at each intervention and control school were invited to participate in data collection at follow-up.

Intervention

A 3-year universal (‘whole of school’) intervention was delivered to all students in grades 8–10. The intervention, based on a pilot study,⁷² involved 16 broad strategies (see box 1) seeking to build the protective factors of students implemented across the three domains of the Health Promoting Schools framework (box 1).⁷³ Each of the 16 broad strategies addressed one or more individual (self-efficacy, problem solving, cooperation/communication, self-awareness, empathy, goals/aspirations) or environmental protective factors (school support, school meaningful participation, community support, community meaningful participation, home support, home meaningful participation, peer caring relationships, prosocial peers). Such protective factors have been found to be correlated with adolescent substance use⁷⁴ and align with a ‘resilience’ approach.^{38–40 75}

Box 1. Intervention and implementation support strategies.

Intervention strategies by Health Promoting Schools domain

Curriculum, teaching and learning

• Age-appropriate lessons (9 hours) on individual protective factors across school subjects (eg, MindMatters⁷⁶ or school-developed curriculum resources)^I,*

• Non-curriculum programmes (9 hours) targeting protective factors (eg, the Resourceful Adolescent Programme)^{77 I,E}

• Additional programme targeting protective factors for Aboriginal students^I,E,*

Ethos and environment

• Rewards and recognition programme^I,E

• Peer support/peer mentoring programmes^I,E

• Antibullying programmes^I,E

• Empowerment/leadership programmes^I,E

• Additional empowerment/leadership/mentoring programmes for Aboriginal students^I,E,*

• Aboriginal cultural awareness strategies^I,E,*

Partnerships and services

• Promotion/engagement of local community organisations/groups/clubs in school (eg, charity organisations)^E,*

• Additional/enhanced consultation activities with Aboriginal community groups^I,E,†

• Promotion/engagement of health, community and youth services in the school^I,E,*

• Additional/enhanced Aboriginal community organisations promoted or engaged^I,E,†

• Referral pathways to health, community and youth services developed and promoted^I,E,*

• Strategies to increase parental involvement in school (eg, school events)^E,*

• Information regarding student protective factors provided to parents via school newsletter^I,E,*

Implementation support strategies

• Engagement with school community including presentations at school staff meetings regarding planned intervention^‡

• Embedded staff support:

  • School intervention officer 1 day a week to support programme implementation

  • Project coordinator to liaise with school sectors and support school intervention officers^§

• School intervention team formed (new team or realignment of existing team, inclusive of school intervention officer and school executive member) to implement intervention

• Structured planning process to prioritise and select appropriate resources/programmes:

  • Needs assessment of student protective factors (when study sample in grade 7 and 9)

  • Two school community planning workshops and one strategy review workshop^§

  • School plan to address intervention strategies endorsed by the school executive

• Intervention implementation guide that described the intervention, planning process, available resources and programmes, tools and templates for intervention implementation.

• Staff mental health training (minimum of 1 hour per school during staff meetings)

• $A2000 per year each for:

  • Teacher release time for intervention implementation or professional development

  • Strategies specifically for Aboriginal students*

• Feedback reports regarding student substance use and protective factors, and intervention implementation (termly)^§

• An Aboriginal Cultural Steering Group was formed comprising Aboriginal staff from local Aboriginal community organisations and Government Departments to provide Aboriginal cultural advice and direction regarding the study design, implementation, evaluation and dissemination

A pragmatic intervention approach⁷⁸ that involved intervention delivery by school staff as a component of routine school practice was adopted to approximate intervention delivery under ‘real-world’ conditions.⁷⁸ Schools were provided with details of existing resources and programmes addressing the 16 broad strategy areas from which they could choose to implement. While schools were required to implement programmes and resources that addressed each of the 16 broad strategies, they had the flexibility to select which specific programme or resource to implement, and the order and manner in which they were implemented. This approach is similar to approaches adopted by previous substance use prevention studies,^{58 61 62} with the exception that selected programmes and resources were not required to have been rigorously evaluated.

To facilitate implementation of intervention strategies, programmes and resources, schools were provided with a comprehensive range of support strategies, including an embedded psychology or education trained implementation support officer; strategies that have been previously reported to facilitate implementation of interventions (box 1).^79–86

Control schools implemented usual school curricula and policies which may have included protective factor strategies and resources similar to, or the same as, those systematically provided to the intervention schools, but were not provided with programme resources or support. A report describing baseline school-level student substance use and protective factor characteristics was provided to control schools.

Data collection procedures

Student demographic and protective factor characteristics and substance use outcomes

Students completed a confidential web-based survey⁸⁷ in class time prior to intervention commencement (baseline: August–November 2011) and immediately following intervention completion (follow-up: July–November 2014). Neither the school staff nor researchers were blind to group allocation.

Implementation of strategies targeting protective factors

To assess intervention implementation by intervention schools,⁸⁸ research staff reviewed school documents and recorded the delivery of intervention strategies monthly. In addition, at follow-up, telephone-based structured interviews were conducted with staff from both groups by interviewers regarding school implementation of intervention strategies and engagement with the intervention during the final year of intervention, School staff from intervention schools were asked their level of engagement with the intervention in the final year.

Measures

Student demographic characteristics

The student survey addressed: age, gender, residential postcode, Aboriginal and/or Torres Strait Islander status, ethnicity and non-English speaking background.

Student substance use

Substance use outcome data were collected using items from an ongoing Australian triennial survey of school students’ health behaviours (see online supplementary appendix A).⁹ Primary outcomes included tobacco (ever and recent) alcohol (ever, recent and ‘risky’) use. Secondary outcomes included marijuana and other illicit substance use. Planned validation of student self-report of smoking via saliva-based cotinine testing^{67 89} was not conducted due to school policies prohibiting drug testing.

Student individual and environmental protective factors

The Resilience and Youth Development module of the California Healthy Kids Survey was used to measure individual and environmental protective factors.⁷⁴ Items for all six individual and three of the environmental factor subscales were selected based on their congruence with the intervention (see online supplementary appendix A). Aggregate individual and environmental protective factor scores were used as secondary outcome measures.

Consistent with a previous study of the survey,⁷⁴ analysis of baseline responses confirmed the subscales were internally consistent and valid (Cronbach's α coefficients: individual 0.55–0.81; environmental 0.77–0.88).⁹⁰ Confirmatory factor analysis⁷⁴ demonstrated the subscale factor structure to be a good model fit (comparative fit index 0.92, root mean square error of approximation 0.04).

Implementation of strategies targeting protective factors

The telephone survey of school staff assessed reported implementation of programs and resources in each of the 16 broad strategy areas (box 1) in during the final year of intervention.

Sample size

The sample size was calculated on the basis of 24 schools (ie, 12 in each group). Based on an assumed parental consent rate of 80%,^{32 91} and loss of students to follow-up from grade 7 to grade 10 of 25%, it was estimated the cohort would consist of 2720 grade 7 students (1360 in each group) and 2040 grade 10 students at follow-up (1020 in each group). Assuming 80% power, a 5% significance level, an intracluster correlation of 0.01⁷² and grade 10 control group prevalence of 14% for recent smoking, 36.2% for recent/risk alcohol use, 25% for marijuana use and 9.3% for other illicit substance use,⁹² the study was estimated to be able to detect an absolute reduction in prevalence of 4.8% for recent smoking, 7.0% for recent/risk alcohol use, 6.2% for marijuana use and 3.9% for illicit substance use in intervention compared with control students.

Statistical analysis

Student demographic characteristics

Student-reported residential postcode was used to calculate student socioeconomic status⁷⁰ and remoteness of residential location.⁹³ Characteristics of students (gender, Aboriginality, socioeconomic status, remoteness, baseline substance use and protective factor scores) completing both baseline and follow-up surveys were compared with those lost to follow-up by logistic regression accounting for potential clustering of students within schools.

Student substance use

Recent tobacco use was defined as having smoked at least one cigarette in the last week, and recent alcohol use as at least one alcoholic drink in the last week (yes/no). The response options for ‘risky alcohol use’ were dichotomised (either ‘none’ or ‘once’/'twice’/'three to six times’/'seven or more times’), as were the response options for both marijuana and other illicit substance use (either ‘none’ or ‘once or twice’/'3–5 times’/'6–9 times’/'10–19 times’/'20–39 times’/'40 or more times’).

Comparison between groups in the prevalence of substance use at follow-up for the cohort grade 10 students in intervention and control schools was undertaken to determine the effectiveness of the intervention using generalised linear mixed models (binomial distribution with a logit link; analysis as treated). All models included a fixed effect for treatment group (intervention vs control) and a random effect for each school to account for clustering of responses within schools. Models were adjusted for a priori selected prognostic variables (age, gender, school type, school size, Aboriginal/Torres Strait Islander status, ethnicity, non-English speaking background, socioeconomic status) and ORs with 95% Wald CIs calculated. Intraclass correlations were estimated on the logistic scale using the methods described by Eldridge et al.⁹⁴

Sensitivity analyses were undertaken according to intention-to-treat principles, where multiple imputation was used to assess the sensitivity of results to missing data under the missing at random (MAR) assumption⁹⁵ from students who were lost to follow-up or changed schools during the intervention period. The method of chained regression equations was used, imputing 10 data sets separately by treatment group and pooling the results using Rubin’s method.⁹⁶ Specifically, this involved a chained regression equations method of generating 10 complete datasets; logistic regression models were used for categorical (binomial, ordinal or multinomial) variables and linear regression models were used for continuous variables. The imputation model included all substance use outcomes, together with all variables that were in the analysis model and treatment group.

Student individual and environmental protective factor scores

Student protective factor subscale scores were calculated by averaging the responses to all items in each subscale. Aggregate individual and environmental protective factor scores were calculated by averaging all relevant subscale scores for each student.⁷⁴ Scores ranged from 1 to 4, with higher scores more favourable.

Linear mixed models were used to assess the effectiveness of the intervention for the aggregate individual and environmental protective factor scores at follow-up. The models included a fixed effect for treatment group (intervention vs control) and a random effect for school to account for clustering of responses within schools. Models were adjusted for the same prognostic variables as per the substance use models. Intraclass correlation was estimated as the proportion of the total variance that is due to between cluster variance.

Implementation of strategies targeting protective factors

Descriptive statistics summarised the number of intervention schools implementing each of the 16 broad intervention strategies that targeted protective factors as identified via project records (intervention years 1–3). Chi-square and t-test analyses examined whether intervention and control schools differed with respect to their reported implementation of protective factor strategies in the final year of intervention.

A criterion for statistical significance of p≤0.05 was used. All analyses were undertaken by an independent statistician using SAS Software V.9.4.⁹⁷

Results

Sample

Schools

Forty-four of the 47 eligible schools were approached prior to achieving the quota of 32 schools (73% consent rate) (see figure 1). Participating schools included 28 government and four Catholic schools. Of the 32 schools, 21 were medium and 11 were large-sized schools. No schools withdrew following allocation.

Figure 1.

Study flow diagram.

Students

At baseline, parental consent was provided for 3530 grade 7 students (76.9% of enrolled students), of which 3115 students participated in the baseline survey (67.9% of enrolled students; 88.2% of students with parental consent). Follow-up data were collected from 2149 of the students who completed the baseline survey (retention rate 69.0%; intervention 67.3%, control 71.6%) with no differential loss to follow-up between intervention and control groups (p=0.1). Reasons for lost to follow-up included: students no longer attending school (n=652; 65.5%), absent from school on follow-up survey days (n=207; 20.8%) or unknown reason for currently enrolled students (n=137; 13.8%). Students who moved between schools (n=30) and those who participated but did not answer substance use items at baseline (n=14) were excluded resulting in a cohort of 2105 students for the primary analysis. All 3115 students who completed the baseline survey were included in sensitivity analyses.

The demographic characteristics of students who completed the baseline survey are shown in table 1. Students who were lost to follow-up compared with those who completed both baseline and follow-up surveys (the cohort) were more likely to report use for each substance use measure (tobacco: ever 17.9% vs 8.1% p<0.01, recent 4.1% vs 1.4% p<0.001; alcohol: ever 37.6% vs 26.8% p<0.01, recent 8.8% vs 4.2% p<0.001, ‘risky’ 8.6% vs 3.7% p<0.001; marijuana: 2.6% vs 1% p=0.003; other illicit substances: 2.0% vs 0.6% p=0.003), and have lower mean individual (2.92 vs 3.04 p<0.001) and environmental protective factor scores (2.88 vs 2.98 p<0.001). Students who were lost to follow-up were also more likely to be Aboriginal and/or Torres Strait Islander (18.1% vs 10.2%, p<0.001). There was no difference for any other demographic characteristics.

Table 1.

Student demographics, substance use and protective factor characteristics of students participating in baseline survey by group (n=3115)

Student characteristics	Intervention n (%)	Control n (%)
Total students	1909	1206
Male	950 (49.8)	607 (50.3)
Age (mean (SD))	12.6 (0.53)	12.6 (0.53)
Aboriginal and/or Torres Strait Islander*	245 (12.8)	151 (12.6)
Socioeconomic status†
Low (<990)	1062 (55.6)	718 (59.5)
High (≥990)	847 (44.4)	488 (40.5)
Remoteness (ARIA)†
Major cities	744 (39.1)	567 (47.1)
Inner regional	565 (29.7)	387 (32.1)
Outer regional/remote	594 (31.2)	250 (20.8)
Ethnicity
Other ethnic, cultural or national origin	235 (12.3)	95 (7.9)
Non-English speaking background
Speak language other than English	119 (6.2)	57 (4.7)
Substance use
Tobacco use—ever	221 (11.7)	124 (10.5)
Tobacco use—recent	49 (2.6)	21 (1.8)
Alcohol use—ever	615 (32.5)	316 (26.7)
Alcohol use—recent	121 (6.4)	53 (4.5)
Alcohol use—‘risky’	111 (5.9)	50 (4.2)
Marijuana use	34 (1.8)	12 (1.0)
Other illicit substance use	23 (1.2)	8 (0.7)
Protective factor score
Individual factors (mean (SD))	2.99 (0.48)	3.03 (0.45)
Environmental factors (mean (SD))	2.93 (0.56)	2.96 (0.55)

*Missing for four students.

†SES and remoteness could not be calculated five students postcode missing (four intervention, one control).

Substance use

Table 1 shows the proportion of students reporting substance use at baseline. There was no difference between intervention and control students for any measure of substance use at follow-up (table 2), with the same result for intention-to-treat sensitivity analyses (see online supplementary appendix B).

Table 2.

Intervention vs control group comparisons at follow-up (n=2105)

Outcome	Intraclass correlations	Intervention group n=1261	Control group n=844	Intervention vs control
Primary outcomes
Substance use		n (%)	n (%)	OR (95% CI)	p Value
Tobacco use—ever*	0.0182	406 (32.5)	235 (27.9)	1.25 (0.92 to 1.68)	0.14
Tobacco use— recent*	0.0280	148 (11.8)	75 (8.9)	1.48 (0.93 to 2.37)	0.09
Alcohol use—ever†	0.0105	770 (61.8)	494 (58.7)	1.11 (0.83 to 1.48)	0.46
Alcohol use—recent‡	0.0149	261 (20.9)	156 (18.6)	1.10 (0.77 to 1.56)	0.60
Alcohol use— ‘risky’§	0.0152	293 (23.6)	196 (23.4)	1.03 (0.74 to 1.43)	0.86
Secondary outcomes
Substance use
Marijuana use¶	0.0163	193 (15.6)	115 (13.7)	1.18 (0.80 to 1.72)	0.39
Other illicit substance use¶	0.0368	85 (6.9)	47 (5.6)	1.42 (0.85 to 2.38)	0.23
Protective factor score		Mean (SD)	Mean (SD)	Mean difference (95% CI)	p Value
Individual**	0.0011	3.02 (0.48)	3.01 (0.49)	−0.01 (−0.07 to 0.06)	0.87
Environmental††	0.0010	2.77 (0.61)	2.76 (0.62)	−0.02 (−0.09 to 0.06)	0.67

*13 missing.

†18 missing.

‡23 missing.

§25 missing.

¶29 missing.

**four missing.

††seven missing.

Student individual and environmental protective factors

Baseline mean individual and environmental protective factor scores are shown in table 1. At follow-up, there was no difference in mean individual or environmental aggregate protective factor scores between intervention and control students (table 2). Similarly, there was no difference between intervention and control students in mean scores for any of the individual or environmental protective factor subscales (see online supplementary appendix C).

School implementation of strategies targeting protective factors

Review of project records across all 3 years of the intervention identified 12 of the 20 intervention schools were recorded to have implemented programmes or resources in each of the 16 strategy areas every year (see online supplementary appendix D for examples of strategies that intervention schools implemented). In each year of the study, either 18 or 19 of the 20 intervention schools were recorded to have implemented programmes or resources in each of the strategy areas.

A total of 232 of the 256 (91%) school staff completed the telephone survey regarding intervention implementation in the final year of the intervention. Comparison of intervention and control schools reported implementation of intervention strategies in the final year of intervention showed intervention schools were more likely than control schools to have incorporated 9 hours of protective factor instruction across at least two school subjects across grade 7–10 (intervention 88% vs control 36%, p<0.01), but not in grade 10 alone (intervention 88% vs control 55%, p=0.08) (see online supplementary appendix E). A higher proportion of Head Teachers at intervention schools reported using resilience resources within curriculum in any grade than control schools (75% and 49%, respectively, p<0.01) and the mean number of resilience resources implemented outside of the classroom was higher in intervention compared with control schools (3.1 and 1.2, respectively, p<0.01). There were no significant differences between intervention and control schools in the reported implementation of the other 15 strategies (see online supplementary appendix E). Between 73% and 84% of intervention school staff reported being moderately or very engaged in the final year of the intervention (Aboriginal contact 73.7% (14/19); Deputy 84.2% (16/19); Head Teacher Welfare 83.3% (15/18); Head Teachers Key Learning Areas 76.4% (68/89)).

Discussion

This study sought to test the effectiveness of a pragmatic intervention delivered by schools on a universal basis that focused on enhancing student individual and environmental ‘resilience’ protective factors as a means of reducing the prevalence of adolescent tobacco, alcohol and illicit substance use. At follow-up, there was no difference in the prevalence of any measure of substance use between intervention and control students, nor was there any difference for aggregate or individual measure of individual and environmental protective factors.

The findings were broadly consistent with evidence from the only other randomised controlled trial of a school-based universal intervention focused solely on promoting the individual and environmental protective factors of adolescent students as a means of reducing substance use.⁴⁶ The intervention in that study was similar to that in the current study in terms of: its pragmatic nature; timing (from grade 8 onwards); duration (3 years); delivery by school staff; strategies (curriculum and school environment) and environmental protective factor content (addressing relationships and meaningful participation at school). However, its content differed in terms of a more limited focus on individual protective factors than the current study.⁴⁶ Despite promising mid-intervention findings for tobacco use favouring an intervention effect, at follow-up the study similarly found no effect of the intervention on tobacco, alcohol or illicit substance use. Additionally, no effect was found for the protective factors measured (school engagement and social relationships), with authors citing insufficient specific intervention content in these areas as a possible explanation.⁴⁶

The hypothesised mechanism of effect for the current study was based on association evidence that an inverse relationship existed between protective factors and substance use.^{25–37 90} As the intervention was ineffective in improving such factors, it remains unknown whether the enhancement of such factors can lead to a reduction in the prevalence of adolescent substance use.

Various aspects of the intervention design may have contributed to the null finding for protective factors. First, the universal nature of the intervention without a targeted intervention for students with lower protective factor scores or with other substance use risk factors may have limited its ability to have a measurable impact. While there is conflicting evidence regarding whether universal, selective or targeted interventions are more effective in reducing substance use,^98–101 the positive findings of one cluster-randomised controlled study undertaken in 43 schools in Hong Kong suggest that an intervention combining both a universal and a targeted approach may be effective. The study reported a positive effect for 8 of 14 targeted protective factors, as well as a reduction in illegal substance use.⁶²

Second, the use of a pragmatic intervention approach allowing school staff to select the type, manner and order of implementation of curriculum resources and programmes may have contributed to the null study findings, as such an intervention approach has been reported to be less likely to be effective than non-pragmatic approaches.^{102 103} Although pragmatic intervention approaches are intended to optimise translation into practice, the potential exists for a loss of intervention efficacy, integrity and fidelity to occur through local selection and adaptation of programmes.^{104 105} The intervention relied, at least in part, on both schools and teachers selecting from a large number of readily available resources and programmes that address resilience protective factors, very few of which are evidence-based, and schools implementing them well. The study findings suggest that the common practice of schools developing and adapting programmes,^22–24 an intervention approach assessed in this trial, may not realise the intended substance use reduction benefits.

Third, the use of programmes and resources that were also accessible to control schools may have contributed to the null findings due to a lack of differential intervention exposure between groups. The likelihood of such an explanation is heightened by the finding of similar strategy implementation levels in both groups at follow-up, with the exception of curriculum-focussed strategies. It is unclear whether contamination with respect to awareness of programmes and resources between intervention and control schools was an issue as it was not specifically assessed; however, the cluster-randomised design at least in part may have reduced this risk.

Fourth, similar to the conclusion of the study by Bond et al,⁴⁶ the duration of the intervention may have been insufficient to impact on student protective factors. As the full intervention was implemented over 2 years (only 2 of 16 strategies were delivered in year 1), the intervention may not have had sufficient time to impact on student protective factors. This possibility is supported by findings from other school-based substance use prevention studies that suggest interventions delivered over 3–4 years rather than 1–2 years may be more effective.¹⁰⁶ Such a conclusion is also supported by a WHO review of evidence regarding the Health Promoting Schools approach that found interventions of longer duration across a range of outcomes were more effective.¹⁰⁷

Finally, three additional design factors may have limited the intervention effect: the intervention’s focus on protective factors only, with no content addressing known risk factors of substance use (such as peer or familial substance use¹⁰⁸); the limited focus on family and community-based protective factors (such as caring parental relationships and meaningful community participation), both of which have been reported to be predictors of substance use¹⁰⁹ and the reported low test–retest reliability of the resilience protective factor measurement tool, which may have led to instability in student responses over time.⁷⁴

Major strengths of this study included the cluster-randomised controlled study design, the use of implementation support strategies and the large sample size. Although the study found, as for school-based research generally,¹¹⁰ a high rate of student attrition (31%), such attrition did not differ between treatment groups and had little impact on the estimated power of the study (difference of 0.3%–0.4%).

Given the significant policy and practice investment in intervention approaches that seek to enhance student protective factors as a means of reducing adolescent substance use, further research is warranted to investigate the effectiveness of this intervention approach. Further research is also warranted regarding whether universal interventions targeting such factors can be effective when augmented with a targeted intervention component either for those students at elevated risk (ie, selective) or those who have already initiated substance use (ie, indicated). Similarly, further research is required to identify intervention approaches that are both capable of being scaled-up to be delivered as part of routine school practice across large populations of secondary schools, and efficacious in reducing adolescent substance use.