Health-Related Outcome Improvements Via the ACCESS Open Minds Youth Mental Health Service Transformation Project in Edmonton, Canada: Amélioration des résultats en matière de santé grâce au projet de transformation des services de santé mentale jeunes d’ACCESS Open Minds/Esprits ouverts à Edmonton, Canada

Abstract

Objective

Growing efforts to transform youth mental health (YMH) services rest on the assumption that they will produce superior outcomes to those of traditional services. We therefore aimed to determine whether implementation of a broad YMH service model in Edmonton, Alberta (the largest site in the pan-Canadian ACCESS Open Minds [AOM] network) resulted in greater improvements in individual-level outcomes compared to a matched control group seen at non-AOM community mental health services.

Method

This retrospective cohort study used data on the Health of Nation Outcomes Scale (HoNOS) collected by trained clinicians, in youth aged 15–25 attending AOM or comparator services for any mental health problem between April 2016 and September 2019. A difference-in-differences approach compared HoNOS outcomes pre- versus post-exposure to the AOM service over a 1-year time horizon, in relation to youth attending a non-transformed service. Propensity score matching and sensitivity analyses were conducted to ensure bias reduction and robustness of observations, respectively.

Results

The number of referrals to AOM Edmonton increased over time (36.3% [95%CI = 11.0%–68.0%] per month), and the site met benchmarks for rapid assessment (within 72 hours) and time to appropriate care (within 30 days). Of 1,078 youth (344 in the intervention [AOM] group and 734 in the control [non-AOM] group) aged 15–25 visiting community mental health centres, the intervention group had statistically greater improvements (7.4% absolute improvement or 1.2× relative improvement) in total HoNOS scores and its behavioural and symptom subscales.

Conclusions

In addition to improving the reach and timeliness of YMH services, AOM Edmonton produced greater improvements in HoNOS total scores (and two of four subscale scores) compared to a matched control group. Comparative evaluations such as these are essential to demonstrating the value of such services and generating continuous cycles of learning and improvement.

Plain Language Summary Title

What are the outcomes of transforming youth mental health services in community settings?

Plain Language Summary

Recent years have seen multiple efforts to transform mental health services for young people. While some evaluations of these have been conducted, surprisingly few have actually used relevant comparison groups. This makes it more difficult to draw conclusions about which direction the field should head in. As a result, we attempted to examine individual outcomes in a major youth mental health service transformation organized around increased capacity and rapid access to appropriate care in Edmonton, Canada. We compared outcomes using the Health of Nation Outcomes Scale (HoNOS) in the transformed ACCESS Open Minds site with traditional community mental health clinics in Edmonton, and found that the numbers of referrals increased while the vast majority of youth received timely assessment and transfers of care within target times. Finally, individuals seen in the ACCESS Open Minds site once it opened had statistically greater improvements in HoNOS scores, suggesting that they received appropriate and effective care.

Introduction

The mental health of adolescents and young adults has become a topic of growing concern both in Canada and around the globe. ¹ Despite widespread recognition that the primary period of risk, onset, and early course of mental illness occurs during the ages of 12–25,^2–4 currently available mental health supports and services have neither the volume nor optimal configuration to meet prevailing needs. ⁵ Because these gaps occur precisely where the system ought to be at its strongest, multiple youth mental health (YMH) service transformations are now underway.

In Canada, one such initiative includes the flagship YMH service transformation project ACCESS Open Minds/Esprits ouverts (AOM), which began in 14 sites across 6 provinces and territories across Canada. AOM was guided by a range of stakeholders, including researchers and clinicians but also decision-makers, partner organizations, and most importantly youth and their families and carers. ⁶ As a research-focused platform funded by the Canadian Institutes of Health Research and the Graham Boeckh Foundation, AOM's goals were to develop, implement and evaluate a principles-based model of service transformation that reduces unmet needs, provides rapid access to appropriate care in a timely manner, encourages engagement of youth and their families or carers, and ensures continuity of services across the age 18 transition. ⁷

Notably, these principles were evaluated in the same way across sites even though they were operationalized in different ways across sites, recognizing the unique structure and nature of each setting.^8–13 Despite this diversity, we have recently validated AOM's primary hypotheses: that across sites there was a 10% increase in referral rates and a 3% decrease in time to an offered evaluation and first service received for each 6-month period after implementation. ¹⁴ To determine relative effectiveness, however, there remains a need to compare outcomes in AOM with standard care. AOM's largest site is in Edmonton, Alberta (metro population 1.3 million), a diverse urban environment where 31% of citizens are under the age of 25, of which only 6% have utilized mental health services.^15,16 We therefore hypothesized that the AOM model's evolving multicomponent intervention package would yield (a) improvements in referral rates and assessment/service delays similarly to the overall project and (b) greater improvements in youth outcomes as compared with standard care.

Methods

Site

A detailed description of the site was previously published as part of a special supplement on AOM implementation. The Edmonton AOM site launched in May 2017 near the city centre, with over 17,000 youth aged 10–24 and a relatively high level of unemployment and low-income families – representing a population with complex needs. ⁸ Led by Alberta Health Services (AHS), the model as implemented in Edmonton began with community mapping and extended to walk-in access to services in a youth-friendly space situated in a downtown YMCA; a well-trained ACCESS Clinician to provide an initial evaluation for a wide range of presentations in an engaging manner within 72 h; youth and family engagement, including peer support; integration of adolescent and young adult teams in order to ensure continuity of care across the age 18 boundary; provision of services appropriate to level and nature of mental health problem within 30 days (a priori benchmark based on recommendations from the Canadian Psychiatric Association), including through the establishment of clear pathways to specialist services such as early psychosis or eating disorders.

Outreach consisted of early identification campaigns to raise public awareness, linking with an expanding set of community resources over time in order to facilitate access, and a multidisciplinary team composition (including psychiatrists, psychologists, social workers, addiction counsellors, peer support workers, occupational and recreational therapists, nurses, family counsellors, and supported employment specialists) with same-day assessment and/or mobile availability outside of the clinic itself. Over time, the model evolved to incorporate single session counselling, weekend clinics with psychiatrists during busy periods, transition visits from health professionals on the inpatient team to ensure integration of those recently discharged, and addictions counselling.

Consistency with the National AOM Project

We recently reported that as hypothesized in the protocol, ⁷ referral rates and wait times to first assessment and first service were measurably improved across the broader pan-Canadian AOM network. ¹⁴ To first assess whether the results in Edmonton were consistent with those found across the network, ¹⁴ we employed a Generalized Additive Model (GAM) with a negative binomial regression to account for overdispersion in count data, fitted using the R package “mgcv” to allow for non-linear relationships. Data from 1,494 youth referred to the Edmonton AOM site over a 41-month period since the program's inception in October 2016 until February 2020, just prior to the onset of the COVID-19 pandemic, allowed us to estimate the change in monthly referral rates for each 1-month progression since the program's launch. The model included a smooth function of time to capture the non-linear trajectory of referrals over time (Supplemental Figure 1). We also added a categorical variable for calendar month to adjust for seasonal effects. The predicted values from the model closely matched the observed values, indicating a good model fit (Supplemental Figure 1).

Outcomes

Sample and Approach

For the examination of outcomes in youth receiving AOM care versus standard care, the target population was that of youth aged 15–25 seeking help for mental health problems (whether these were deemed to meet the threshold criteria required for a DSM diagnosis or otherwise). Within this, the intervention group was youth within the same age range who sought care from AOM beginning with its launch on 6 April 2017. Any patient obtaining care at AOM from that date until 30 September 2018 was followed for up to one year (i.e., until 30 September 2019). Outcomes during this 1-year period were examined alongside outcomes in the same individuals during the year before the AOM site opened (from 6 April 2016). The before‒after difference in outcomes in the AOM group was then compared with similar before‒after differences in all youth of the same age range who had at least one visit to a non-AOM community mental health service during the same time periods. Per AHS’ designation, community mental health centres are those which provide “outpatient assessment, treatment and follow-up for individuals experiencing addiction and/or mental health concerns. Clinicians work with individuals across the lifespan, and families where appropriate, to assess and develop collaborative treatment goals and service plans.” ¹⁷ They included walk-in options along with “short-to-moderate term outpatient counselling; education, skills and support group programs in community and clinic; and referral to other programs and resources.” ¹⁷ In multiple ways, then, their mandate is similar to that of AOM.

Ethics

Approval to conduct this study was granted by the University of Alberta's Health Research Ethics Board, in addition to ethics approval for the overall AOM project from the Douglas Research Centre.

Outcome Measure

Common scales and routine clinical data collected across multiple sites under the aegis of AHS were harnessed to evaluate outcomes of these differential interventions. The study cohort (including both AOM participants and their non-AOM comparators) was identified as youths aged 15–25 who received care in Edmonton-based community mental health centres (AOM and other) via administrative databases held by AHS.^18,19 The Health of Nation Outcomes Scale (HoNOS) is a widely used scale for routine assessment of mental health outcomes in several countries, including in Canada,^20–22 and has been implemented for child and YMH services in Edmonton by AHS since 2016. ¹⁸ Its developers aimed to measure the health and social functioning of people with persistent mental health problems over time and in response to the receipt of services.^23–26 Evaluations of the HoNOS have been conducted in many settings and have indicated good predictive validity, adequate or moderate to good reliability, sensitivity to change, and acceptability in routine use.^27–29

HoNOS scores are based on a rating of the 12 items measuring specific behaviours: (1) Overactive, aggressive, disruptive or agitated behaviour; (2) Non-accidental self-injury; (3) Problem drinking or drug-taking; (4) Cognitive problems; (5) Physical illness or disability problems; (6) Problems associated with hallucinations and delusions; (7) Problems with depressed mood; (8) Other mental and behavioural problems; (9) Problems with relationships; (10) Problems with activities of daily living; (11) Problems with living conditions and daily domestic routine; and (12) Problems with occupation and activities in daytime. Each item is scored as 0–4, with 0 indicating no problem and 4 representing a severe, incapacitating or unacceptable problem. ³⁰ Change in HoNOS scores is usually reported as a percentage change in the mean value for individuals and total scores, section scores over time, or as the percentage of patients whose difference in total score increased, decreased, or remained unchanged.^24,25,31,32

Data Sources

The HoNOS measures themselves were conducted at each site by similarly trained clinicians, with the first (pre-) measure collected at the initial visit to the service. Assessments were conducted by clinicians, and took place either in person or virtually, depending on the mode of service delivery. At each timepoint, ratings integrated all information available from clinical interviews and case reviews regarding the most severe problem experienced during the previous two weeks.

To calculate interval change in the HoNOS score, the last measure within 1 year of the initial visit date was compared with the initial measure. We considered dates of the first visit to the AOM clinics after 6 April 2017 as the index-dates for exposed youth with mental health problems (intervention group). Dates of the first visit to the non-AOM clinics after 6 April 2017 were the index-dates for unexposed youths with mental health problems (control group). Patients were excluded from the analysis if the interval change in HoNOS score could not be calculated or if their data were lacking in either or both measures: for example, if seen only in emergency or crisis settings, due to unanticipated discharges such that a follow-up HoNOS score was unavailable, or receiving only brief therapy with no follow-up.

Statistical Analyses

First, sample characteristics were derived using descriptive statistics. The included (analytic: all with HoNOS scores pre- and post-treatment) and excluded (only HoNOS scores pre-treatment) samples were compared using simple t-tests.

Second, the association between the AOM intervention and HoNOS measures was determined via a difference-in-difference approach to HoNOS score changes between the intervention and control groups. Specifically, we compared the pre‒post change in HoNOS score using the following formula ³³ :

I = (Hi1 − Hi2) − (Hc1 − Hc2)

where I represents impact, Hi1 represents the HoNOS score for the intervention group pre-treatment, Hi2 represents the HoNOS score for the intervention group post-treatment, Hc1 represents the HoNOS score for the control group pre-treatment, and Hc2 represents the HoNOS score for the control group post-treatment. A positive impact (improvement) indicates a reduction in the HoNOS score which is assumed to be associated with the intervention. ³⁴

To reduce bias due to differences in demographic and clinical characteristics between intervention and control groups, we applied a Kernel propensity score matching technique^35,36 through which an exposed youth with a mental health problem was matched with many unexposed youths by the nearest propensity scores within a calliper of 0.001. Controls with smaller distances were given larger weights. Propensity scores were estimated by a logistic regression where the dependent variable was the likelihood for a patient to be selected in the intervention group; the independent variables were sex, age, year of visit, the most frequent reason to see doctors (e.g., mental health disorder vs. others) based on diagnostic billing code, type of visit (assessment, consultation, or treatment), type of encounter (intake, clinic, or multidisciplinary visit), number of encounters (visits), in-hospital death, comorbidity (based on the Charlson comorbidity index)^37,38 and severity of disease as measured by initial HoNOS score (sub-clinical, mild, moderate, or severe). ¹⁸ Balance tests were performed to examine the goodness of matching (whether bias was reduced by the matching). ³⁹

Besides total HoNOS score, we performed a subgroup analysis for each of the four HoNOS subcategories including behaviour, impairment, symptoms, and social functioning. We conducted a similar analysis a priori for each item in order to report on which specific elements the AOM intervention did or did not impact significantly. Statistical significance was set to P < .05. All analyses were undertaken using Stata SE 16.0 (www.stata.com).

Reporting

Bearing in mind the STROBE reporting guidelines, ⁴⁰ we used the STROBE reporting checklist (included in supplementary materials) when editing this article.

Results

Findings Regarding Primary AOM Hypotheses

On average, the number of referrals to AOM increased by 36.3% (95%CI = 11.0%‒68.0%) per month from program implementation [estimated first derivative of the smooth function 0.311 (SE = 0.106)]. Among youth who were offered an initial evaluation, the vast majority (97.5%) were offered an appointment within 3 days of the first successful contact [95% CI = 96.6%–98.3%]. Among those who received service(s), 97.2% received these within 30 days of the first in-person appointment (95% CI = 96.3%–98.2%).

HoNOS Outcomes

Sample Characteristics

Of the initial sample of 670 individuals in the AOM (intervention) group and 2,696 individuals in the non-AOM (control) group, 326 (48.7%) and 1,962 (72.8%) were excluded, respectively, because they lacked post-treatment HoNOS scores. The analytic sample therefore became 1,078 youths aged 15–25 visiting community mental health centres between 6 April 2017 and 30 September 2018 who had at least two HoNOS measures (before and after one year of the index date). Of this, the intervention (AOM) group included 344 patients (32%) and the control (non-AOM) group included 734 patients (68%) (Table 1).

Table 1.

Characteristics of Patients in AOM and Control Groups.

Characteristic	AOM	Control	P-value
N (#)	344	734	1078
Male, n (%)	155 (45.1)	309 (42.1)	.36
Age, mean (SD)	20.6 (2.1)	21.3 (2.1)	.000
Index year of 2018, n (%)	122 (35.5)	212 (28.9)	.029
Charlson comorbidity index			.001
1, n (%)	64 (18.6)	80 (10.9)
2+, n (%)	11 (3.2)	13 (1.8)
In-hospital death, n (%)	1 (0.29)	5 (0.68)	.422
Number of visits to CMHC pre-inter., mean (SD)	9.2 (14.3)	5.7 (18.5)	.001
Encounter types
Multidisciplinary visits to CMHC pre-inter., mean (SD)	5.5 (10.3)	2.4 (5.6)	.000
Clinic visits to CMHC pre-inter., mean (SD)	1.6 (3.0)	0.8 (2.1)	.000
Intake visits to CMHC pre-inter., mean (SD)	0.3 (0.6)	0.1 (0.3)	.000
Visit types
Consultation visits to CMHC pre-inter., mean (SD)	4.9 (5.2)	2.2 (4.6)	.000
Treatment visits to CMHC pre-inter., mean (SD)	10.9 (20.1)	6.1 (13.8)	.000
Diagnosis: MH disorder is the most frequent in claims, n (%)	298 (86.4)	456 (62.1)	.000
Severity by HoNOS scoresa
Sub-clinical, n (%)	3 (0.9)	11 (1.5)	.397
Mild, n (%)	30 (8.7)	78 (10.6)	.331
Moderate, n (%)	45 (13.1)	155 (21.1)	.002
Severe, n (%)	266 (77.3)	490 (66.8)	.000

^a Sub-clinical: all items rated 0 or 1; mild: at least one item rated 2, but all items rated 2 or less; moderate: one item rated 3 or 4; severe: two or more items rated 3 or 4.

Of note, there were no significant differences in age or sex at birth between the included (analytic) and excluded samples (Supplementary Table A1). Other than the HoNOS impairment subscale, the included sample for both AOM and non-AOM groups had higher HoNOS total and subscale scores than the excluded sample (Supplementary Table A2), signifying a more complex and/or severe presentation than their excluded counterparts.

Multiple sociodemographic and clinical characteristics differed between the intervention and control groups (Table 1): those who attended the AOM clinic were significantly younger, more likely to visit clinics in 2018, and more likely to have more comorbidities (all Ps < .05). Notably, the group later referred to AOM had greater numbers of all visit and encounter types prior to the AOM clinic's opening, greater frequency of mental health (ICD) disorder diagnoses across visits (86.4% of patients in the intervention group and 62.1% of patients in the control group), and included more HoNOS-defined severe patients than the control group (77.3% vs. 66.8%) (all P < .05).

Propensity Score Matching

Results from the multivariate logistic regression (Table 2), which was used to estimate propensity scores for matching AOM and non-AOM groups, showed statistically significant differences still remained in several characteristics including age, index year, Charlson comorbidity index, intake visits, consultation visits, whether a mental health problem was the most frequent billing code when visiting physicians, and HoNOS-defined moderate severity. These differences indicated a need for further matching along such dimensions prior to comparing HoNOS outcomes.

Table 2.

Logistic Regression on the Likelihood of Being Treated (to Estimate Propensity Scores).

Covariate	Odds ratio	P-value	95% CI
			Low	High
Male	1.02	.915	0.759	1.361
Age	0.83	.000	0.773	0.887
Index year of 2018	1.99	.000	1.450	2.740
Charlson comorbidity index	1.63	.002	1.192	2.223
In-hospital death	0.57	.625	0.059	5.495
Number of visits to CMHC pre-inter.	0.99	.146	0.968	1.005
Encounter types
Multidisciplinary visits to CMHC pre-inter.	1.05	.003	1.017	1.087
Clinic visits to CMHC pre-inter.	0.94	.137	0.866	1.020
Intake visits to CMHC pre-inter.	2.19	.000	1.581	3.041
Visit types
Consultation visits to CMHC pre-inter.	1.10	.000	1.059	1.145
Treatment visits to CMHC pre-inter.	1.01	.201	0.996	1.020
Diagnosis: MH disorder is the most frequent	3.13	.000	2.159	4.537
Severity by HoNOS score: severe reference
Sub-clinic	0.52	.346	0.135	2.016
Mild	0.70	.154	0.425	1.145
Moderate	0.55	.004	0.363	0.822
Constant	5.04	.029	1.177	21.595
Number of observation	1078
LR chi2 (12)	222.79
Prob > chi2	0.000
Pseudo R2	0.165
Log likelihood	−563.64

Once conducted, propensity matching using the variables for which differences existed significantly reduced bias (Table 3): per the balance test, R-squared was reduced from 16.4% (P = .000) to 2.2% (P = .144); mean bias was reduced by 73% (from 25.6 to 6.8); median bias from 23.4 to 5.0; and Rubins’ B from 100.9 to 34.6.

Table 3.

Balance Test Results.

Sample	Pseudo R2	LR chi2	P > chi2	Mean bias	Median bias	B	R
Unmatched	0.164	221.51	.000	25.6	23.3	100.9	1.34
Matched	0.022	20.77	.144	6.8	5	34.6	0.63

Total HoNOS Scores

Initial (pre-treatment) and subsequent (at re-assessment) HoNOS scores, and the differences between the AOM and control groups after propensity score matching, are shown in Table 4. Before treatment, the average HoNOS score was 16.27 in the AOM group and 15.52 in the control group. After up to 1 year of treatment, these numbers were 8.98 and 9.71, respectively, representing improvements of 7.29 points (44.8%) in the AOM group and 5.80 points (37.4%) in the control group. The improvement of those receiving care in the AOM arm was 7.4% (44.8%-37.4%) absolutely or 1.2x (44.8%/37.4%) relative to the control arm; this difference was statistically significant.

Table 4.

HoNOS Scores Pre and Post Treatment in AOM and Control Groups After Matching.

				Difference in difference
	Measure	AOM	Control	Mean	P-value
Total score
HoNOS pre	Points	16.27	15.52
HoNOS post	Points	8.98	9.71
Pre-post difference	Points	7.29	5.80	1.49	<.05
	%	44.8%	37.4%	7.4%
	Times			1.2
Sub-scale
Total score for behaviour (Q1–3)
HoNOS pre	Points	3.68	3.17
HoNOS post	Points	1.65	1.83
Pre-post difference	Points	2.03	1.34	0.68	<.01
	%	55.1%	42.4%	12.7%
	Times			1.3
Total score for impairment (Q4–5)
HoNOS pre	Points	0.88	1.36
HoNOS post	Points	0.55	0.98
Pre-post difference	Points	0.34	0.38	−0.05	>.05
	%	38.2%	28.1%	10.0%
	Times			1.4
Total score for symptom (Q6–8)
HoNOS pre	Points	6.34	5.76
HoNOS post	Points	3.38	3.48
Pre-post difference	Points	2.97	2.28	0.69	<.01
	%	46.8%	39.6%	7.2%
	Times			1.2
Total score for social (Q9–12)
HoNOS pre	Points	5.36	5.23
HoNOS post	Points	3.41	3.43
Pre-post difference	Points	1.96	1.80	0.16	>.05
	%	36.5%	34.4%	2.1%
	Times			1.1

HoNOS Subgroup Analyses

Subgroup analyses showed that behavioural and symptom categories were differentially impacted compared to impairment and social categories. Before treatment, the average HoNOS behavioural category scores were 3.68 (AOM) and 3.17 (control). After 1 year of treatment, these numbers were 1.65 and 1.83, respectively, reflecting an improvement of 2.03 points (55.1%; AOM) and 1.34 points (42.4%; control), respectively, an absolute gain of 12.7% (relative gain 1.3×).

For the symptom category, the average HoNOS score was 6.34 (AOM) and 5.76 (control) before treatment. After 1 year of treatment, these numbers were 3.38 and 3.48, reflecting improvements of 2.97 points (46.8%; AOM) and 2.28 points (39.6%; control), respectively (7.2% absolute improvement, 1.2x relative improvement).

For impairment and social categories, the difference in differences was not statistically significant, despite the AOM (intervention) group having higher pre- and lower post-treatment HoNOS scores than the control group.

Discussion

This is the first evaluation of comparative outcomes of the pan-Canadian ACCESS Open Minds model compared to standard care, and one of the first comparative evaluations of YMH service transformation more broadly. ⁴¹ In this large urban centre, the AOM model resulted in more referrals over the period post-implementation, with the majority receiving rapid assessments and access to a first service as per a priori benchmarks. Indeed, the reach and timeliness of AOM in Edmonton were comparable and even superior to the outcomes for the multi-site AOM study as a whole, while demonstrating a greater improvement in important dimensions of HoNOS total scores (and two of four subscale scores) compared to a matched control group.

Partly because the need for multiple (pre- and post-) outcome measures required that individuals were followed in services for some time, the final (analytic) sample consisted of a subset of individuals who were retained in care, with correspondingly moderate-to-severe illness profiles. Even here, however, individuals followed at AOM had on average higher (more impaired) scores at entry than those in the non-AOM group, while their post-treatment scores were lower (less impaired). This suggests that individuals entering AOM had more persistent treatment needs which following AOM became less intensive, indicating that the AOM Edmonton service and intervention package was noninferior to standard care, even while it increased referral intakes and rapid assessments/transitions to appropriate services.

Of course, it has been argued that a statistically significant change may not be meaningfully different.^42,43 While there are (to the best of our knowledge) no widely established thresholds or minimally meaningful difference for HoNOS, Parabiaghi et al. ⁴⁴ attempted to identify a ‘reliable and clinically significant change’ that would represent a meaningful outcome. Their model included two criteria for adequate change (at least 4 or 8 points change in total HoNOS scores), and cut-offs for clinically significant change (a total score of 10 was the threshold between mild and moderate illness, 13 between moderate and severe illness). Of note, our total HoNOS scores in AOM and non-AOM groups changed by ∼7.3 and ∼5.8 points respectively – suggesting that the AOM intervention was (at a minimum) noninferior to standard care, with both groups moving from a severe to a mild status on average.

Strengths of the current study include the common data infrastructure harnessed for the analyses, and propensity score matching to ensure that AOM and control subjects’ HoNOS scores were comparable. A decision was also made to close the assessment period in late 2019, in order to avoid the confounding impact of the COVID-19 pandemic on study outcomes. Limitations include the fact that HoNOS assessments were conducted as part of regular routine outcome measures instead of at each timepoint; this means that individuals without follow-up HoNOS scores (representing a less severely affected subgroup) were excluded, that the included sample reflected a more moderate-to-severely affected population, with its conclusions then generalizable only to that specific subgroup. Furthermore, it is possible that the AOM service valued data and measurement to a greater extent than the non-AOM service, explaining why there was a greater proportion of (or more positive) HoNOS completions at the former site. Finally, while we have previously noted the progressive nature of the AOM service transformation, ⁸ the current evaluations were conducted over the entirety of the project period and cannot distinguish between the effect of individual site-level adaptations on HoNOS outcomes. While an evolving and iterative application of principles is a strength of the AOM transformation, it becomes challenging to disentangle the extent to which these findings are attributable to the principles themselves versus the site-specific operationalization of those principles.

Observational methods such as DID approaches may certainly mask unintended selection effects, such as in the group who continued to receive services across pre- and post-periods. Had HoNOS scores been possible to capture for individuals whose condition improved enough that they did not return to services, HoNOS improvements at the non-AOM site might on average have paralleled (or even exceeded) those who transitioned to the AOM site. However, if the observed improvements for those who transition to the AOM site were truly greater than those who remained at the non-AOM site, this could be for varying reasons: the effectiveness of the individual treatments provided, their relative intensity and combination, and/or the configuration and ethos (represented by a philosophy of community-based outreach and flexibility) of the novel services themselves and their roots in the early intervention movement.

Finally, outcomes should also be examined in the context of resource allocation. Since one of the major objectives of early intervention paradigms is to shift care upstream, from intensive and acute/emergency settings to community services, then it will also be of importance to examine the specific health services utilized and associated costs in comparison with traditional settings. This aspect of the AOM research agenda will be the subject of additional reports, and will enable the field to draw conclusions that have wider pertinence to the growing field of YMH.