An observational study of adolescent health outcomes associated with school‐based health service utilization: A causal analysis

Abstract

Objective

Our aim is to examine the unbiased association between use of school‐based health services (SBHS) and student health outcomes.

Data Sources

Data are from a nationally representative health and well‐being survey of 8500 New Zealand high school students from 91 high schools.

Study Design

Student data were linked to the level of SBHS available to them: no SBHS, regular clinics from visiting health professionals, a health professional onsite, or a health team onsite.

Data Collection/Extraction Methods

Causal analyses are used to compare utilization of SBHS and their association with student‐reported health outcomes, including foregone health care, depressive symptoms, emotional and behavioral difficulties, suicide risk, substance use, and unsafe sexual behaviors.

Principal Findings

Results from the multinomial propensity score–weighted regressions show that the use of SBHS was associated with poorer health outcomes, suggesting that selection bias was present due to unmeasured confounders. Instrumental variable analyses found that that students using team‐based SBHS had a 4.7 percent (95% CI 0.5‐8.9) probability of high levels of depressive symptoms compared to 14.2 percent (95% CI 11.5‐16.8) among students not using team SBHS. For suicide attempt, students using team‐based SBHS had a 2.0 percent (95% CI −0.3‐4.2) probability of a suicide attempt in the previous 12 months compared to 5.6 percent (95% CI 2.6‐8.5) among students not using team SBHS.

Conclusions

These analyses suggest that team‐based SBHS are associated with better mental health among students who attend them.

1. INTRODUCTION

School‐based health services (SBHS) are health services located in schools which ideally can provide youth‐appropriate health care through their accessible, low‐cost, youth‐focused services and comprehensive care.1, 2 Most SBHS provide a range of services from acute and primary care to mental health, sexual health, substance use counseling, and health promotion. SBHS are among the few broadly available health services specifically for young people, and their numbers are increasing globally.3 As such, SBHS have the potential to improve the health and well‐being for large numbers of young people.

However, to date the evidence of their effectiveness is mixed.4, 5 There have been no cluster‐randomized controlled trials of SBHS. Observational studies face hurdles such as selection bias in that students using SBHS often face higher adversity and higher health needs than students not using SBHS.6, 7, 8 It is challenging to match schools providing health services with control schools, such that the pertinent comparison is the presence of the SBHS and not differences between the schools in terms of their student sociodemographic characteristics.9

In New Zealand, there is a great deal of variability in the availability and funding of school health services. In well‐resourced SBHS, students are able to drop in to the clinic for any health concerns, ranging from acute medical conditions to longer‐term chronic medical conditions, mental health issues, sexual and reproductive health or other health concerns. Students are able to be referred to the school clinic by pastoral care staff or teachers when they are concerned about the students’ physical or mental health. Funding of SBHS in NZ high schools varies considerably based on the socioeconomic status (SES) of the community in which the school is located, with schools in low‐SES communities eligible to receive direct government funding to provide comprehensive SBHS. Schools in higher SES communities have had to make either do with funding from their own school budget or rely on health services from visiting public health nurses, who are often limited to one ½‐day clinic per week. This results in a great deal of variability of school health services, ranging from well‐resourced teams of medical, nursing, and other health professionals on‐site for most of the school week to visiting nurse clinics of a couple of hours per week, or, in some schools, no health services at all beyond basic first aid. To date, observational studies of SBHS have largely ignored the variability between schools in the level of health services provided and have tended to look at the presence or absence of health services between schools.3, 10

This study aims to examine the association between utilization of SBHS and self‐reported health outcomes among students. The outcomes considered include foregone health care, mental health concerns, suicidality, substance use, and sexual health behaviors. We use data from a large nationally representative survey of 8500 high school students from 91 high schools in New Zealand and link data on these health outcomes to information on the level of SBHS in each school. Students self‐report utilization of school health services and also a range of health concerns. We aim to estimate the causal association between using SBHS and five health outcomes. These relationships are likely confounded by selection bias, whereby students who have health concerns are more likely to use SBHS. To try to mitigate any such selection bias, the relationship between SBHS utilization and health outcomes is analyzed from two perspectives. Firstly, multinomial propensity scores are used to estimate the unbiased association between SBHS utilization and health outcomes using observed covariates. Propensity scores can help reduce confounding, including selection bias, by balancing on observed covariates. This assumes “strong ignorability” whereby given the observed covariates, the treatment outcome is independent of the potential outcomes. Ideally a broad range of covariates influencing SBHS utilization would be used, but as our data are cross‐sectional, covariates were restricted to those unable to be influenced by health care utilization, such as age, sex, ethnicity, and socioeconomic status. To address these shortcomings, a second approach utilizing the variability between schools in terms of availability of SBHS is also used. Under this approach, SBHS is considered as an intervention at the school level. We first use “intention‐to‐treat” (ITT) analyses assuming all students are potential users of SBHS, whether they have used SBHS. If SBHS are truly randomly implemented, this approach will provide an unbiased estimate of the “treatment.” However, ITT estimates of the impact of SBHS are conservative as most students do not use SBHS, and any treatment effect of students using SBHS is diluted among those who do not use SBHS. We therefore also use instrumental variable analyses with the availability of SBHS in each school as the “instrument” to estimate the effect on health indicators among students using SBHS.

2. METHODS

Data for this study come from Youth'12, a nationally representative health survey of New Zealand high school students conducted in 2012.11, 12 Youth'12 is part of an ongoing project to monitor the health and well‐being of secondary school students in New Zealand through a series of cross‐sectional surveys conducted every 5 to 6 years. Sample size calculations for this survey aimed to give reasonable prevalence estimates of health indicators among the four main ethnic groups in New Zealand.11 A two‐stage cluster design was utilized to select a nationally representative sample. In total, 8500 randomly selected high school students from 91 randomly selected high schools completed the survey, accounting for 3 percent of the total 2012 high school roll in New Zealand. Table 2 shows the demographic characteristics of the sample, which, apart from a slightly higher percentage of female students, is similar demographically to the national population of high school students.11

Response rates for schools and for students were 73 percent and 68 percent, respectively. The most common reasons for students not participating were being absent from school or involved in other school activities on the survey day. The survey was administered using Internet tablets. No keyboard data entry was required; questions and answers could also be heard through headphones, and responses were made by touching the screen.13 Written consent was obtained from each participating school and student. Information was sent home to all families, and parents could opt to have their child excluded from the survey. Ethics approval was gained from the University of Auckland Human Participants Ethics Committee (ref 2011/206).

2.1. School‐based health service data

All schools that participated in Youth'12 were invited to participate in a SBHS survey after the student survey had been conducted in 2012. Of the 91 schools that participated in Youth'12, one school had subsequently closed. All the remaining 90 schools agreed to take part in the health services survey and provided information on the level of SBHS in their school.

School leaders were asked about the level of SBHS in their school with the question “What level of service best describes your school health service?” with the response options “First aid and urgent health care,” “Regular health clinics from visiting health professionals,” “Approximately one health professional onsite for most of the week,” and “A health team on‐site for most of the school week.” Schools with only first aid services were classified as having no SBHS. These differing levels of SBHS reflect resources available in each school for health services, and there are marked differences in terms of hours of clinician time per 100 students, level of training of clinicians, infrastructure, and collaboration with the wider school.14

2.2. School‐based health care utilization

Utilization of school‐based health services was assessed from students’ responses to two questions in the Youth'12 survey: “When was the last time you went for health care?” with response options “0‐12 months ago,” “12‐24 months ago,” and “More than 2 years ago”; and “Which of the following places have you used for health care in the past 12 months (you can choose as many as you need)?” with response options that included “School health clinic.” Students who responded that they had received health care in the previous 12 months and had used the school health clinic were then categorized into one of three groups based on the level of SBHS available in their school (health clinics from visiting health professionals, on‐site health professional, or team SBHS).

2.3. Student health measures

Health outcomes were selected a priori to reflect the major health and well‐being issues among young people in New Zealand.15 These included foregone health care, mental health concerns, suicidality, substance use, and sexual health behaviors (Table 1).

Table 1.

Description of health measures and student and school characteristics

Health measures	Description of measures
Foregone health care	Students were asked “In the last 12 months, has there been any time when you wanted or needed to see a doctor or nurse (or other health care worker) about your health, but you weren't able to?” with response options “Yes” or “No.” The percentage of students who had foregone health care in the previous 12 mo was 18.6%.
Depressive symptoms (10 items)	Students’ depressive symptoms were assessed by the Reynolds Adolescent Depression Scale‐Short Form (Mean 19.56, SE 0.07, range 10‐40, Cronbach's alpha 0.90). This is a well‐validated instrument for measuring depression symptoms in community adolescent samples and has been demonstrated to have acceptable reliability and validity across ethnic groups within New Zealand.22, 33
Emotional and behavioral difficulties (20 items)	The Strengths and Difficulties Questionnaire (SDQ) is a well‐validated screening instrument for child and adolescent mental health issues.34 The total difficulties score of the SDQ is derived from summing the 20 items relating to emotional symptoms, conduct problems, hyperactivity‐inattention, and peer problems. Previous research has shown that higher self‐rated total difficulties score are associated with greater psychopathology.23 The mean of the 20‐item total difficulties score was 11.4 (SE 0.06), range 0‐40.
Suicidality (three items)	We derived a combined measure to assess suicidality including suicidal ideation, planning, and behaviors. Students were asked the following questions: “During the last 12 months have you seriously thought about killing yourself?”; “During the last 12 months have you made a plan about how you would kill yourself?”; and “During the last 12 months, have you tried to kill yourself (Attempted suicide)?” The suicidality measure was the mean of response scores (1 = “Not at all,” 2 = “Not in the last 12 months,” 3 = “Once or twice,” 4 = “Three or more times”) to the three questions. The mean of the three items was 1.27 (SE 0.01), range 1‐4, Cronbach's alpha 0.84.
Unsafe sexual health (four items)	Unsafe sexual health was assessed by a combined measure of four questions, including no condom use the last time they had sex, no contraception the last time they had sex, never or sometimes contraception use, and never or sometimes condom use. The mean of the four dichotomous items was 0.08 (SE 0.004), Cronbach's alpha 0.71.
Substance use (three items)	Substance use was a combined measure of binge alcohol use (five or more alcoholic drinks within 4 h), occasional or more often cigarette use, and weekly or more often marijuana use. The mean of the three dichotomous items was 0.12 (SE 0.005), Cronbach's alpha 0.59.
Student demographics
Age, Sex, Ethnicity	Students self‐reported their age, sex, and ethnicity. Age was grouped into students 15 y of age and younger and students 16 y of age and older. The standard New Zealand census question was used to assess ethnicity: “Which ethnic group do you belong to? (you may choose as many as you need).” Forty‐two percent of participants identified with more than one ethnic group. To facilitate statistical analyses, discrete ethnic groups were created using the ethnicity prioritization method, by assigning students to one ethnic group in the following order: Maori, Pacific, Asian, Other, and NZ European ethnicities.
Socioeconomic deprivation	The socioeconomic status of each student was measured by nine indicators of socioeconomic deprivation: access to resources at home (car, phone, computer), the number of rooms and people in their household (overcrowding), how frequently they have moved house, not enough money for food, no holidays, garages or living rooms used as bedrooms, and parents who they live with being unemployed. Based on previous latent class analyses,35 students who reported two or more indicators of socioeconomic deprivation were classified in the socioeconomic deprivation group.
Geographical location	During the survey, students were asked to provide their home address so as to ascertain the mesh block in which they lived and residential location. The mesh block is a small‐area census unit of about 100 households. The categories were main urban (cities, major urban areas, or large regional centers with a minimum population of 10 000 people), minor urban (urbanized settlements with a population between 1000 and 9999 people), and rural (rural centers and locations with populations less than 1000 people).
School characteristicsa
School gender	Coeducational, boys only, or girls only
School funding type	Private, integrated, or state‐funded
Size of student roll	Less than 399, 400‐799, 800‐1199, or 1200 and more
School decile	Decile 1 schools represent the 10% of schools with the highest proportion of students from low‐SES communities whereas decile 10 schools are the 10% of schools with the lowest proportion of these students.

^aStructural characteristics of schools were obtained from the New Zealand Ministry of Education.

2.4. Analyses

Descriptive statistics on student and school characteristics were compared between students not using SBHS and those using the three levels of SBHS using chi‐square tests for categorical variables. We focus on six primary outcomes: foregone health care (N = 8414), depression symptoms (N = 8182), emotional and behavioral difficulties (N = 8188), suicide risk (N = 8326), sexual health risk (N = 8219), and substance use (N = 8220). As there were less than 10 percent missing data for the primary outcomes, no attempt was made to impute data for missing responses. Two sets of analyses were performed, one adjusting for student covariates (age, sex, ethnicity, socioeconomic status, and geographical location) and the second adjusting for both student‐ and school‐level covariates (school funding, type of school, school size, and school socioeconomic decile band). These student‐ and school‐level covariates are detailed in Table 1. The analyses compare multinomial propensity scores (MPS) and intention‐to‐treat (ITT) and instrumental variable (IV) regression estimates. To account for these multiple comparisons, we consider findings statistically significant only at P values <0.01.

2.5. Multinomial propensity score analyses

The MPS regression compares differences in health indicators between students who have used SBHS and students who have not used SBHS. Multinomial propensity score analyses are used to compare the four levels of SBHS. As our data are from a two‐stage cluster design with unequal probability of selection, we follow the recommendation of DuGoff16 and incorporate survey weights into the propensity score models and in the final regressions use a combined weight by multiplying the propensity score weights and our sampling weights.

Covariates were selected based on previous analyses looking at factors associated with SBHC utilization17 and outcome variables15 and included age, gender, ethnicity, socioeconomic deprivation, and urban vs rural location. Generalized boosted regression (GBM) was used to estimate propensity scores for multiple treatment groups. This uses an iterative process with multiple regression trees to capture complex and nonlinear relationships between treatment assignment and the pretreatment covariates without overfitting the data.18 As we are interested in the effects of SBHS on the entire population of students, we estimate average treatment effects with inverse probability of treatment weighting (IPTW). We use the procedure recommended by McCaffrey et al18 for average treatment effects, whereby each student's probability of accessing differing levels of SBHC is weighted to match the entire sample. One advantage of GBM for estimating propensity scores is that the process can be fine‐tuned to balance covariates based on decision rules. We achieved balance across all five covariates using procedures that minimized the absolute standardized mean difference, with all covariates being less than 0.2. In addition, we conducted 60 pairwise comparisons of covariates across the four levels of SBHS. Only two showed absolute standardized mean difference greater than 0.1. The covariates with the highest absolute standardized mean difference were urban/rural location (0.12) comparing health clinics from visiting health professionals to team SBHS, reflecting the difficulties small communities face providing comprehensive school health services.

To estimate the overlap between levels of SBHS and the propensity score, box plots of the propensity scores for each possible SBHS were examined. This showed good overlap among groups and nonzero probability of receiving each level of SBHC. The final step of the MPS analyses uses the IPTW weights multiplied by the sample weights to weight an OLS regression predicting the six health indicators. In sensitivity analyses (not shown), all five covariates were added to these regressions, a form of “doubly robust” estimation, which did not change the results substantively.

2.6. Instrumental variable analyses

The instrumental variable approach is used to address the issue of endogeneity from selection bias, whereby variables associated with health outcomes influence treatment assignment. The “instruments” used in these analyses are the availability of school‐based health services in each school. The models are just identified with one instrument for each level of SBHS utilization. The analyses can be thought of as two‐stage least‐squares regressions, where in the first stage, use of one of the three levels of SBHS is regressed on the instrument (availability of SBHS) along with student covariates (X represents a vector of student‐level covariates) and school‐level covariates (Z represents a vector of school‐level covariates). Assuming instrument validity, the predicted value of using one of the three levels of SBHS represents the exogenous (random) use of SBHS.

$${\text{Usedvisitingsbhc}}_i={\pi }_0+{\pi }_a{X}_i+{\pi }_b{Z}_i+{\pi }_1\text{visitsbhc}+e_i$$

$${\text{Usedonsitesbhc}}_i={\pi }_0+{\pi }_a{X}_i+{\pi }_b{Z}_i+{\pi }_2\text{onsitesbhc}+e_i$$

$${\text{Usedteamsbhc}}_i={\pi }_0+{\pi }_a{X}_i+{\pi }_b{Z}_i+{\pi }_3\text{teamsbhc}+e_i$$

To assess whether availability of the three types of SBHS were not weak instruments, we used partial F tests. The null hypothesis was that there was no association between availability of SBHS and students’ utilization of SBHS. An F statistic over 10 suggests the instrument is not weak.19 In models adjusting for student covariates, the partial F statistic for students using health clinics from visiting health professionals was F(10, 81) = 8.55, for using an onsite health professional was F(10, 81) = 21.55, and for using team school‐based health services was F(10, 81) = 41.88. For models adjusting for student‐ and school‐level covariates, the partial F statistics were F(19, 72) = 7.31, F(19, 72) = 21.6, and F(19, 72) = 67.0, respectively. These results suggest that utilization of visiting health professionals may be a weak instrument and that caution is required in interpreting subsequent IV estimates relating to visiting health professionals.

In the second stage, each health concern is regressed on the predicted values of using a school health service for each level of SBHS.

$$\begin{array}{cc}\hfill \text{Health}{\text{outcome}}_i=& {\beta }_0+{\beta }_a{X}_i+{\beta }_b{Z}_i+{\beta }_3{\text{Usedvisitingsbhc}}_i\hfill \\ & +{\beta }_4{\text{Usedonsitesbhc}}_i+{\beta }_5{\text{Usedteamsbhc}}_i+e_i.\hfill \end{array}$$

We follow Angrist20 and assume linear models for all health indicators and the utilization of SBHS as linear in the IV analyses. These equations are estimated via limited‐information maximum likelihood using the “cmp” command in STATA (StataCorp. 2015; Stata Statistical Software: Release 14. College Station, TX: StataCorp LP).21 All analyses use sampling weights and account for the clustering of students within schools.

3. RESULTS

3.1. Student and school characteristics

Overall, 1383 (16.9 percent) students had used school‐based health services in the previous 12 months, comprising 344 (4.1 percent) students who had used school‐based health clinics staffed by visiting health professionals, 637 (7.9 percent) students who used school‐based health clinics staffed by an on‐site health professional, and 402 (4.9 percent) students who had used a school‐based health clinic staffed by a team of health professionals (Table 2). There were marked sociodemographic differences between students who had used the different types of school‐based health services. Most of these differences reflected the availability of the different types of SBHS. For example, students from families experiencing socioeconomic deprivation were more likely to use team‐based SBHS than other types of SBHS. But this was due to team‐based SBHS being more prevalent in schools with high numbers of families experiencing socioeconomic deprivation (Table 3). Utilization of SBHS was highest among students attending schools with a health team on‐site (26.6 percent) and in schools with a health professional on‐site for most of the school week (27.8 percent), compared to schools with visiting health professionals (9.2 percent).

Table 2.

Prevalence of used school health services by student and school characteristics

	N	%	Has not used SBHS,No. (%)	Used health clinics from visiting health professionals,No. (%)	Used on‐site health professional,No. (%)	Used team‐based SBHS,No. (%)	P value
No of students	8500		6879	344	637	402
Age (y)
≤15	5489	64.6	4482 (65.2)	216 (62.9)	405 (63.3)	239 (59.5)	0.27
≥16	3000	35.4	2387 (34.8)	128 (37.1)	232 (36.7)	163 (40.5)
Gender
Female	4623	54.3	3623 (52.6)	206 (59.9)	443 (68.8)	277 (68.9)	0.008
Male	3874	45.7	3254 (47.4)	138 (40.1)	194 (31.2)	125 (31.1)
Ethnicity
Asian	1051	12.4	859 (12.6)	19 (5.6)	68 (10.6)	56 (13.9)	<0.001
Maori	1701	20	1346 (19.6)	112 (32.5)	137 (21.7)	49 (12.2)
Pacific	1201	14.3	891 (13.1)	25 (7.4)	63 (9.8)	187 (46.5)
Other	511	6	406 (5.9)	22 (6.5)	39 (6.1)	22 (5.5)
NZ European	4024	47.3	3367 (48.8)	166 (48)	330 (51.8)	88 (21.9)
SES deprivation
No	6791	80	5571 (81.1)	261 (75.5)	520 (81.7)	252 (62.7)	<0.001
Yes	1709	20	1308 (18.9)	83 (24.5)	117 (18.3)	150 (37.3)
Geographical location
Major city	6320	74.7	5072 (74.1)	183 (53.8)	494 (77.3)	380 (94.5)	<0.001
Minor city	946	11	766 (11.1)	91 (26.4)	54 (8.4)	6 (1.5)
Rural	1234	14.3	1041 (14.8)	70 (19.7)	89 (14.3)	16 (4)
School funding
Private	533	6.1	405 (5.7)	0	48 (7.5)	51 (12.7)	—
Integrated	980	11.3	767 (10.9)	30 (8.5)	92 (14.3)	71 (17.6)
State	6987	82.6	5707 (83.4)	314 (91.5)	497 (78.2)	280 (69.7)
Type of school
Coed	6225	72.7	5156 (74.4)	221 (63.8)	425 (65.9)	280 (69.7)	0.44
Boys	795	9.8	619 (9.4)	50 (14.7)	34 (6.4)	5 (1.2)
Girls	1480	17.5	1104 (16.1)	73 (21.5)	178 (27.7)	117 (29.1)
School size
≤399	998	10.7	813 (10.8)	83 (23)	23 (3.5)	25 (6.2)	0.33
400‐799	1641	19.4	1308 (19.1)	110 (32.4)	126 (19.5)	69 (17.1)
800‐1199	1856	22.4	1561 (23.3)	69 (20.4)	142 (23.2)	42 (10.4)
≥1200	4005	47.5	3197 (46.9)	82 (24.2)	346 (53.8)	266 (66.2)
School SES decile
Deciles 1‐3	1793	20.9	1326 (19.1)	94 (27)	99 (15.3)	51 (12.7)	0.02
Deciles 4‐7	3296	39	2803 (40.9)	187 (54.3)	197 (31.7)	71 (17.6)
Deciles 8‐10	3411	40.1	2750 (40)	63 (18.6)	341 (53)	280 (69.7)

Table 3.

Student and school characteristics by availability of SBHS

	No school health servicesN = 858	Regular health clinics from visiting health professionals,N = 3786	Approximately one health professional on‐site for most of the school week, N = 2294	Health team on‐site for most of the school weekN = 1545	P value
	%	%	%	%
	10	44.0	27.6	18.4
Age
Less than 15 y	42.2	34.2	34.6	35.9	0.27
Over 15 y	57.8	65.8	65.4	64.1
Gender
Female	40.4	52.5	57.9	60.6	0.47
Male	59.6	47.5	42.1	39.4
Ethnicity
Asian	31	6.4	13.4	15.6	<0.001
Maori	10.2	24.1	21.8	12.3
Pacific	6	4.7	10.8	46.8
Other	9.1	5.3	6.7	5
NZ European	43.7	59.5	47.3	20.3
Socioeconomic deprivation
No	87.6	83.9	82.1	63.3	<0.001
Yes	12.4	16.1	17.9	36.7
Geographical location
Major city	91.4	59	80.4	95.4	<0.001
Minor city	1.4	19.8	6.8	0.8
Rural	7.1	21.2	12.8	3.8
School funding
Private	34	‐	4.8	7.8	—
Integrated	17.4	9.4	11.3	12.8
State	48.6	90.6	83.9	79.4
Type of school
Coed	59.6	74.5	70	79.4	0.56
Boys	30.6	9.4	8.4	1.5
Girls	9.9	16.1	21.5	19.1
School size
≤399 students	25.4	14.7	3.1	4.2	0.18
400‐799	5.9	28.6	13.4	13.6
800‐1199	38.1	22.3	25.1	10.2
≥1200 students	30.6	34.4	58.4	72
School SES decile
Decile band 1‐3	—	16.1	14.3	52.8	—
Decile band 4‐7	17.6	50.7	40.4	20.8
Decile band 8‐10	82.4	33.2	45.4	26.4
Used SBHS	—	9.2	27.8	26.6

3.2. Multinomial propensity score estimates

The IPTW weighted regression models for the association between use of SBHS and health indicators are presented in Table 4. The results show higher levels of health concerns among students using SBHS. For example, depressive symptoms were worse among students using health clinics from visiting health professionals compared to students not using SBHS. Similar findings were found for emotional and behavioral difficulties, suicide risk, substance use, and unsafe sexual health behaviors.

Table 4.

Health indicators by utilization of SBHS—multinomial propensity score estimates

	Foregone health care		Depressive symptoms (RADS‐SF)		Emotional and behavioral difficulties		Suicide risk		Substance use		Unsafe sexual health behaviors
	Mean	β (95% CI)	Mean	β (95% CI)	Mean	β (95% CI)	Mean	β (95% CI)	Mean	β (95% CI)	Mean	β (95% CI)
Has not used SBHS	0.17	Ref	19.32	Ref	11.2	Ref	1.25	Ref	0.12	Ref	0.07	Ref
Used regular health clinics	0.29	0.27 (0.05, 0.49)	20.98	1.66a (0.36, 2.96)	12.79	1.62a (0.47, 2.76)	1.42	0.17a (0.06, 0.28)	0.18	0.07b (0.03, 0.1)	0.15	0.08a (0.03, 0.13)
Used on‐site SBHS	0.24	0.04 (−0.12, 0.21)	21.09	1.77b (1.25, 2.29)	12.71	1.54b (0.87, 2.21)	1.33	0.08a (0.02, 0.14)	0.18	0.06b (0.03, 0.09)	0.12	0.05 (0.01, 0.09)
Used team‐based SBHS	0.24	0.07 (−0.12, 0.26)	19.76	0.44 (−0.35, 1.22)	11.80	0.62b (0.28, 0.95)	1.25	0.004 (−0.06, 0.07)	0.13	0.01 (−0.04, 0.06)	0.08	0.01 (−0.02, 0.04)

^a P < 0.01.

^b P < 0.001.

3.3. ITT and IV estimates of SBHS

Table 5 shows the ITT and IV estimates of the associations between level of SBHS and health indicators. The ITT associations show that among students in schools with team‐based health services, there were lower levels of depressive symptoms, emotional and behavioral difficulties, and suicide risk compared to students in schools with no school‐based health services. The IV estimates are similar to the ITT associations, but higher in magnitude as they account for whether students actually utilized these services. For example, the ITT estimates of predicted depression symptoms were 18.8 (95% CI = 18.4‐19.2) among students in schools with team‐based health services compared to 20.2 (95% CI = 19.8‐20.7) among students in schools without SBHS. The corresponding IV estimates of predicted depression symptoms among students actually using team‐based SBHS were 14.8 (95% CI = 12.8‐16.7) compared to 20.2 (95% CI = 19.8‐20.6) among students not using SBHS.

Table 5.

Intention‐to‐treat and instrumental variables estimates of the causal association between type of SBHS and health indicators

	Health team on‐site for most of the school week		Approximately one health professional on‐site for most of the school week		Regular health clinics from visiting health professionals
	ITT β (95% CI)	IV β (95% CI)	ITT β (95% CI)	IV β (95% CI)	ITT β (95% CI)	IV β (95% CI)
Outcome = foregone health care
Student covariates	−0.02 (−0.05, 0.02)	−0.06 (−0.19, 0.07)	−0.02 (−0.05, 0.01)	−0.07 (−0.19, 0.05)	0.002 (−0.03, 0.03)	0.05 (−0.32, 0.41)
Student & school cv	−0.03 (−0.06, −0.003)	−0.13 (−0.23, −0.03)	−0.03 (−0.06, −0.002)	−0.11 (−0.21, −0.02)	−0.008 (−0.03, 0.02)	−0.11 (−0.39, 0.17)
Outcome = depressive symptoms
Student covariates	−1.44 (−2.12, −0.75)b	−5.41 (−8.12, −2.69)b	−0.68 (−1.35, 0.001)	−2.34 (−4.76, 0.09)	−0.54 (−1.24, 0.17)	−5.51 (−13.3, 2.28)
Student & school cv	−1.02 (−1.73, −0.31)b	−4.05 (−7.09, −1.00)	−0.42 (−0.99, 0.16)	−1.57 (−3.68, 0.54)	−0.37 (−0.97, 0.23)	−4.36 (−11.3, 2.60)
Outcome = emotional and behavioral difficulties
Student covariates	−0.77 (−1.42, −0.13)	−2.87 (−5.28, −0.46)	0.09 (−0.51, 0.69)	0.42 (−1.76, 2.59)	0.24 (−0.35, 0.82)	2.97 (−3.64, 9.58)
Student & school cv	−0.86 (−1.47, −0.25)	−3.38 (−5.76, −1.02)a	0.03 (−0.42, 0.47)	0.03 (−1.52, 1.58)	0.08 (−0.40, 0.55)	0.71 (−4.61, 6.02)
Outcome = suicide risk
Student covariates	−0.10 (−0.17, −0.03)a	−0.38 (−0.64, −0.12)a	−0.07 (−0.13, −0.002)	−0.23 (−0.46, −0.001)	−0.04 (−0.10, 0.02)	−0.41 (−1.13, 0.31)
Student & school cv	−0.11 (−0.18, −0.03)a	−0.41 (−0.70, −0.12)a	−0.07 (−0.13, −0.01)	−0.26 (−0.49, −0.03)	−0.05 (−0.11, 0.01)	−0.62 (−1.35, 0.12)
Outcome = substance use
Student covariates	−0.01 (−0.04, 0.02)	−0.04 (−0.15, 0.06)	0.01 (−0.02, 0.03)	0.03 (−0.05, 0.11)	0.01 (−0.02, 0.03)	0.12 (−0.15, 0.40)
Student & school cv	−0.01 (−0.04, 0.02)	−0.05 (−0.17, 0.08)	0.01 (−0.02, 0.03)	0.03 (−0.06, 0.12)	0.01 (−0.02, 0.04)	0.10 (−0.21, 0.41)
Outcome = unsafe sexual health
Student covariates	0.02 (−0.01, 0.04)	0.06 (−0.02, 0.14)	0.02 (0.003, 0.03)	0.06 (0.01, 0.11)	0.02 (0.003, 0.03)	0.20 (0.04, 0.36)
Student & school cv	0.02 (−0.001, 0.03)	0.06 (−0.01, 0.13)	0.02 (0.01, 0.03)	0.07 (0.02, 0.12)a	0.02 (0.001, 0.03)	0.17 (−0.02, 0.36)

Abbreviations: cv—covariates; ITT—intention‐to‐treat; IV—instrumental variable.

^a P < 0.01.

^b P < 0.001.

Foregone health care was lower among students in schools with team‐based and on‐site SBHS, but these associations did not reach statistical significance (P < 0.01). Apart from unsafe sexual behaviors, there were no significant ITT or IV associations between SBHS with a health professional on‐site or regular clinics from visiting health professionals with any of the health indicators. Conversely, unsafe sexual behaviors were higher, not lower, among students attending clinics with an on‐site health professional.

To further analyze the significant effects of team SBHS, discrete outcomes were created from the continuous mental health variables. Established cut‐points for the RADS‐SF22 and total difficulties score23 were used to describe clinically relevant groups of students with high levels of depressive symptoms and with emotional and behavioral difficulties. Any suicide attempt in the previous 12 months was used to group students on their level of suicide risk. Instrumental variable analyses using probit regression were then used to predict the probabilities of these outcomes among students using team school‐based health services compared to students not using team school‐based health services. Adjusting for student sociodemographic variables, students using team‐based SBHS had a 4.7 percent (95% CI 0.5‐8.9) probability of high levels of depressive symptoms compared to 14.2 percent (95% CI 11.5‐16.8) among students not using team SBHS. For emotional and behavioral difficulties, students using team SBHS had a 4.0 percent (95% CI −0.2‐8.3) probability of high levels of emotional and behavioral difficulties compared to 13.0 percent (95% CI 11.4‐14.5) among students not using team SBHS. For suicide attempt, students using team‐based SBHS had a 2.0 percent (95% CI −0.3‐4.2) probability of a suicide attempt in the previous 12 months compared to 5.6 percent (95% CI 2.6‐8.5) among students not using team SBHS.

4. DISCUSSION

The aim of this study was to examine the effect of using SBHS on health indicators using observational data. The main issue when examining this association is the problem of selection bias, in that students who have health concerns are more likely to use SBHS than students who do not have health concerns. We use two modern statistical approaches to address this confounding. Firstly, we use MPS to balance observed covariates across treatment groups. Secondly, we use analyses that treat the availability of SBHS as essentially random after controlling for student‐level and school‐level covariates (ITT and IV). Results from the MPS analyses indicated that the use of SBHS was associated with poorer health outcomes, suggesting that selection bias was still present. This is probably due to the limited number of covariates available to estimate the propensity score, with unmeasured factors present that influenced both the use of SBHS and health outcomes, therefore violating the assumption of strong ignorability. Previous research has shown that effect estimates from propensity score analyses are biased when the set of measured covariates is not sufficient to address strong ignorability.24

In contrast, results from the ITT and IV analyses suggest that team‐based SBHS are associated with fewer depression symptoms, fewer emotional and behavioral difficulties, and lower suicide risk among students using these services. On‐site SBHS by a health professional and SBHS from visiting health professionals did not show similar positive associations with these mental health concerns. The IV estimates were able to gauge the magnitude of the mental health benefits among students using team‐based SBHS. These were clinically significant; students using team SBHS had one‐third the mental health concerns that students not using SBHS had.

Under assumptions of constant treatment effects, the IV estimates are average treatment effects. However, it is unlikely that all students respond similarly to team‐based SBHS. Under weaker assumptions of monotonicity, that is, no defiers (meaning there are no students who receive team‐based SBHS in schools without team‐based SBHS), the IV estimates are the local average treatment effect among the compliers, or treatment among the treated.25 While it is difficult to know exactly who the complier group is, just over one‐quarter of students in schools with team‐based health services used SBHS. It is important to note that students may be utilizing team‐based SBHS for health conditions other than mental health issues so the complier group is likely smaller than the total proportion of students utilizing SBHS and so our estimates are likely conservative.

In New Zealand, team‐based SBHS often includes pastoral care teachers (school guidance counselors), social workers, and youth workers working together with medical and nursing staff to address emotional and behavioral issues among students. This “wraparound” model of health service provision may well be better suited than single providers to addressing the often complex psychosocial issues faced by adolescents experiencing emotional distress.26 Furthermore, team‐based SBHS are associated with higher levels of staffing, with higher levels of medical and nursing time per week, than on‐site SBHS by a health professional and SBHS from visiting health professionals.14 Being able to address emotional and behavioral concerns requires time without the pressures of needing to see as many students as possible in the clinic time available.

While one previous study27 has documented reduced cigarette and marijuana use among school health clinic users, we, like other studies,28 found no association between utilization of SBHS and student substance use. We also found little evidence of improved sexual health behaviors; in fact, in schools with an on‐site health professional, there was increased risk of unsafe sexual health behaviors among students. That finding raises the possibility that the availability of SBHS cannot be regarded as random but reflects underlying sexual health risk among populations attending these schools. Any confounding between unsafe sexual behaviors and aspects of schools with on‐site health professionals that are not accounted for by the student‐level and school‐level covariates used in our analyses would violate the assumptions behind the ITT and IV analyses.

Valid ITT and IV estimates rely on the assumption that there is no relationship between the instrument, in this case availability of SBHS, and student health other than through utilization of SBHS. This includes the assumption of no confounding between health concerns and SBHS. Previous research from the United States has shown that SBHS are more likely in schools where students experience more health risks or where state or national policies support such services.29, 30 SBHS in New Zealand are distributed according to funding and location, with SBHS being more available in lower socioeconomic schools, which receive extra funding, and more available in schools in larger cities. We attempted to account for this through student‐ and school‐level covariates, but it is arguable how successful these variables were in making treatment assignment “strongly ignorable.”31 There also might be an association between other school factors that we did not measure, such as quality of school leadership, which may influence student well‐being as well as enabling higher levels of SBHS. These concerns raise questions around the validity of school health services as an appropriate instrument. From a research perspective, ideally SBHS would be randomly assigned at the school level and future research should consider cluster‐randomized trials of SBHS. Another assumption of the ITT and IV analyses is that the effect of SBHS on an individual student does not influence availability or impact of SBHS for other students, the so‐called SUTVA.32 This is unlikely to be the case in our situation as students accessing SBHS are likely to discuss with their peers their experiences of SBHS, thereby violating the no‐interference assumption. It is unclear how these interactions would influence our results.

At the same time, the lack of positive findings with respect to unsafe sexual health behaviors and substance use is concerning and suggests that SBHS in New Zealand may need to focus more on these areas.

5. CONCLUSIONS

The evaluation of school health services using observational data is complex. Selection bias confounds the association between use of school health services and health outcomes, as students with health concerns, especially mental health concerns, are more likely to use school health services. We attempted to address this issue by treating SBHS as a random school‐level intervention. While questions remain about whether the provision and level of these SBHS are truly random, our findings suggest that team‐based school health services may well be associated with better mental health outcomes. However, questions remain about baseline confounding, and future studies should consider cluster‐randomized studies with the provision of school health services randomized between schools.

Supporting information

 

Denny S, Grant S, Galbreath R, Utter J, Fleming T, Clark T. An observational study of adolescent health outcomes associated with school‐based health service utilization: A causal analysis. Health Serv Res. 2019;54:678–688. 10.1111/1475-6773.13136