Naturalistic evaluation of a sport-themed mental health and wellbeing app aimed at men (MindMax), that incorporates applied video games and gamification

Abstract

Introduction

While men display lower help-seeking rates than women, there is a lack of mental health interventions targeting men. To address this issue, we developed a smartphone app named MindMax, an Australian Football League (AFL)-themed app containing psychoeducational modules teaching strategies derived from positive psychology and acceptance and commitment therapy. MindMax also incorporates gamification, casual video games, and social connection and is intended to appeal to male Australians interested in AFL. This study reports results from a naturalistic trial intended to investigate whether using MindMax was associated with improved wellbeing, resilience, and help-seeking intentions.

Methods

We conducted a naturalistic trial from July 2017 to May 2018, where participants were given access to MindMax to use as they wished, and asked to answer wellbeing surveys at multiple time points. As we employed a customised version of the General Help-Seeking Questionnaire (GHSQ), we conducted an exploratory factor analysis and extracted two factors that we interpreted as ‘personal help-seeking’ and ‘impersonal help-seeking’. Mixed design MANOVAs were conducted with flourishing, resilience, personal help-seeking, impersonal help-seeking, relatedness, and sense of connection (self-group overlap) to the MindMax community to assess change between Day 1–30 and Day 1–60.

Results

313 participants (174/313, 55.6% female; 131/313, 41.9% male) completed the survey at baseline and at least one follow-up survey. We observed significant 30-day and 60-day increases in impersonal help-seeking intentions and sense of connection to the MindMax community, and 60-day increases in flourishing. 30-day increases in sense of connection were highest in our male participants with high base wellbeing, present in our female participants, and not present in our male participants with low base wellbeing. 60-day increases in sense of connection were higher in high-wellbeing participants than in low-wellbeing participants.

Discussion/conclusion

Our findings are encouraging as they could be attributed to participants' exposure to MindMax. However, they could also be attributed to other factors that may also have motivated trial participation. Future research can consider investigating more explicitly the role of conformity to masculine norms and how that may affect uptake of mHealth technologies and help-seeking behaviour.

Highlights

• •MindMax combines wellbeing modules, video games, and a social community.
• •Factor analysis of the GHSQ resulted in two factors: personal and impersonal help-seeking intentions.
• •We observed 30- & 60-day increases in impersonal help-seeking intentions and sense of connection to the MindMax community.
• •We also observed 60-day increases in flourishing.
• •Overall results suggest MindMax was positively received by its users.

1. Introduction

1.1. Men's mental health and help-seeking

Previous research has identified that Australian men have mental health experiences and needs that are different to those of Australian women, including prevalence of mental health problems (Rice et al., 2018), reduced help-seeking (Ellis et al., 2014), and being less comfortable with structured Internet health interventions (Batterham and Calear, 2017). Pressures to uphold traditional masculine values, such as stoicism, toughness, aggression, and self-sufficiency, act in tandem with multiple factors, such as higher mental health stigma, lower mental health literacy, and poorer emotional competence (Corrigan and Watson, 2007; Ellis et al., 2014). These contribute to lower rates of help-seeking for mental health problems in Australian men compared to women (Ellis et al., 2013; Rice et al., 2018). Some research has shown that younger Australian men in particular are comfortable managing and seeking help for their mental health anonymously and from informal sources like the Internet (Ellis et al., 2013). There is a need for targeted initiatives (including awareness campaigns and intervention programmes) proactively aimed at men to facilitate their help-seeking for mental health problems (Rice et al., 2018).

Resilience, or the ability to adapt to stressful situations, has been positively linked to higher levels of psychological wellbeing (Souri and Hasanirad, 2011). In a similar vein, poorer mental health is linked to lower levels of seeking help for mental health problems (Rickwood et al., 2005). Recent research shows that Australian males who behave according to rigid, traditional views of masculinity are more likely to display unfavourable mental health outcomes, including higher rates of negative affect and suicide ideation (Rice et al., 2018; The Men's Project and Flood, 2018), and lower rates of help-seeking (Seidler et al., 2016; Wong et al., 2017). A meta-analysis by Wong et al. (Wong et al., 2017) suggests this may be attributable to three dimensions of conformity to masculine norms in particular: self-reliance, power over women, and being a playboy. While they found strong associations between these dimensions and increased negative mental health, decreased positive mental health, and decreased help-seeking, they found no such associations for the other two dimensions: primacy of work and risk-taking (Wong et al., 2017). This points towards the multifaceted nature of masculinity and suggests that offering more varied, healthy, and flexible alternatives of masculinity, and role models championing these alternatives, may be fruitful (Schlichthorst et al., 2018; The Men's Project and Flood, 2018). According to evidence-based recommendations, mental health interventions tailored to men should be informal and action-based (as opposed to talk-based) and focused on building strengths (as opposed to reducing deficiencies) (Ellis et al., 2012; Ellis et al., 2013). Furthermore, couching such interventions within subcultures men are already present in reduces the burden on them to engage in help-seeking actions that many men may perceive as being a threat to their masculinity (Rice et al., 2018), especially those with poorer mental health outcomes.

Drawing heavily from these recommendations, the University of Sydney and Queensland University of Technology partnered with the Australian Football League Players' Association (AFLPA), The Mind Room, and the Young and Well Cooperative Research Centre to build a mobile phone application named MindMax. MindMax is an AFL-themed app that combines psychoeducation, social connection, and applied games (including gamification and casual games), aimed at men aged 16 to 35 years who are interested in the Australian Football League (AFL) and/or video games. While MindMax is aimed at encouraging access to mental health information and minimising barriers to help-seeking for younger men, it is also intended to be used by everyone.

1.2. App overview

MindMax is made up of three main components: psychoeducational (wellbeing training modules), social (a community feed), and play (a goal-kicking casual game named Flick Footy). These components are supplemented by gamification, in which Flick Footy costs ‘footies’ (in-app currency) to play, and these ‘footies’ are earned through completing training modules and interacting with other users through posting or commenting in the community feed. The wellbeing training modules are based on positive psychology as well as acceptance and commitment therapy principles and cover a range of topics, including values, mindfulness, and strategies for dealing with unhelpful thoughts and emotions. We have previously written a more in-depth description of MindMax's structure (Cheng et al., 2018), as well as its applied games specifications and design rationale (Mitchell et al., 2017).

Participants who were representative of MindMax's intended end users were able to influence its features and design specifications throughout its development process through participatory design workshops and regular user testing (Cheng et al., 2018). Finally, MindMax was updated regularly across its development life with additional content, as well as performance and design fixes (Cheng et al., 2018).

1.3. Study aim

This study aimed to investigate whether we would observe a change in study participants' flourishing, resilience, help-seeking intentions, and/or sense of connection to the MindMax community across time, and whether gender and base wellbeing influenced any such change. This change was evaluated with two separate mixed design MANOVAs, one comparing baseline scores with 30-day scores, and the other comparing baseline scores with 60-day scores. In a mixed design MANOVA, with time as the within-subject factor and gender and base wellbeing as the between-subjects factors, a significant change would be indicated by a significant main effect of time, and/or significant higher-order interaction effects between time and one or more between-subjects factors.

2. Methods

2.1. Ethical approval

This study received ethical approval from The University of Sydney's Human Research Ethics Committee (Protocol No. 2016/652).

2.2. Design

Mohr et al. (2015) argue that in today's rapidly changing technological landscape, it is impractical to evaluate static versions of apps and other behavioural intervention technologies, as these interventions run the risk of becoming obsolete after often lengthy evaluations. Instead, the authors recommend evaluating the intervention principles that drive the technology, while reporting all changes made to the intervention. For these reasons, and to accommodate MindMax's development schedule, we conducted a single-arm, naturalistic longitudinal trial to evaluate the ongoing development of MindMax.

Once ready, the Minimum Viable Product (MVP) version of MindMax was simultaneously released for the public and deployed into trial. Data from this trial and other associated data (including usage analytics and user testing) was used to direct MindMax's maturation into a more developed product. As described in our previous publications (Cheng et al., 2018), content and performance updates were made until February 2018, including additional psychoeducational modules, new casual games, aesthetic improvements, trophies and team functionality, and a ‘Flick Footy Max’ competition in December 2017, where a Playstation 4 Pro and a MindMax-themed football were offered as prizes to users who scored the highest in the ‘Flick Footy’ casual game. We hence applied multiple updates to MindMax across the duration of the trial with the aim of improving its delivery of intervention principles.

2.3. Participants

We recruited participants from 14 July 2017 to 28 February 2018, according to these inclusion criteria: aged 16–35 years, resident in Australia, and having access to a smartphone or other smart device that could access the Internet and run MindMax. Participants were recruited via paid and unpaid advertising on social media (Facebook, Twitter, and Instagram) and posters displayed on and around the authors' affiliated institutions. While study advertising varied in wording across social media platform and physical medium, all advertising introduced MindMax as a wellbeing app, asked participants for their help to research wellbeing, and informed participants that they would receive vouchers for their time and feedback. All advertisements also directed potential participants to a website specifically set up to act as a portal to the study. This website once again explained what participation in the study would involve before linking to the baseline (Day 1) survey, and was deactivated following the close of participant recruitment.

In early stages of recruitment, a minority of participants were also recruited directly from the MindMax app via a banner overlay on the social feed.

2.4. Procedure

The naturalistic longitudinal trial ran from 14 July 2017 to 9 May 2018. After screening, obtaining informed consent, and presenting the baseline (Day 1) survey (through the study portal website described in Section 2.3), we directed participants to the MindMax homepage and asked them to “download MindMax, create an account, and use it as you wish”. This was done as MindMax was managed and distributed by the AFLPA. Importantly, as MindMax was also made publicly available at the same time, trial participants could interact with both other trial participants as well as the organic user base (i.e. those who came to begin using the app by means other than recruitment into the naturalistic trial). More information on the organic user base is reported by Vella et al. (2018).

We specified that when registering for MindMax, participants should use the same email address they had used to complete the survey. Email addresses from the baseline survey were checked against those in the MindMax database and sent a reminder to register for a MindMax account if the participant had not done so within three days of completing baseline. Only those email addresses who could be matched with an email address in the MindMax database were invited to complete follow-up surveys (Day 30, Day 60, Day 90, and Close of Study). However, due to low sample sizes, we report only the first three time points (Day 1, Day 30, Day 60) in this paper. We sent up to two reminder emails, with three-day waits in between, if the participant did not respond to the initial survey invitation email. Participants were reimbursed AU$10 for each survey completed as a recognition of their time and effort spent participating in the study.

2.5. Measures

We collected participant demographics at baseline (Day 1) and self-reported wellbeing, resilience, and help-seeking intentions at Days 1, 30, and 60. Specifically, the following measurements were collected:

• •Mental wellbeing, through the 14-item Warwick-Edinburgh Mental Wellbeing Scale (WEMWBS) (Tennant et al., 2007).
• •Psychological flourishing, through the seven-item Flourishing scale (Diener et al., 2010).
• •Resilience, through the 10-item Connor-Davidson Resilience scale (CD-RISC 10) (Campbell-Sills and Stein, 2007).
• •Help-seeking intentions, through the personal-emotional items of the General Help-Seeking Questionnaire (GHSQ) (Wilson et al., 2005). Following the recommendations of the scale creators, who encourage modification of the GHSQ to fit study aims (Deane and Wilson, 2007), these items were further adapted as follows: the items for ‘teacher’, ‘pastor/priest’, and ‘youth worker’ were not administered, and an item for ‘someone online, who you don't know personally’ was added, as such people have recently emerged as a common source of help (Ellis et al., 2013). We also added a ‘MindMax’ item, and an ‘Other’ item with free text input, coding back responses to the relevant items if required, leading to a total of 11 items.
• •Sense of connection to the MindMax community, through the one-item Assessment of Self-Group Overlap (Schubert and Otten, 2002), which we adapted by replacing the word ‘group’ with the word ‘MindMax’. This question presents participants with seven images which vary in representations of social distance. A value of ‘1’ indicates the furthest distance between the self and the MindMax community and a value of ‘7’ indicates total overlap of the self and the MindMax community.

Table 1 presents the range and validation means and standard deviations (SDs; if available) of each scale. All reported means and SDs are taken from their original validation studies (which were performed with non-Australian samples), with the exception of the WEMWBS where we present Australian population data (Davies et al., 2016). These means and SDs are hence presented only to provide a frame of reference. With the exception of the adapted GHSQ and the adapted Assessment of Self-Group Overlap, all scales used have established strong reliability.

Table 1.

Scale range and validation means/SDs.

Scale	Range	Validation mean (SD)
WEMWBSa	14–70	52.8 (6.8)
Flourishing	8–56	44.97 (6.56)
CD-RISC 10	0–40	27.21 (5.84)
GHSQ (adapted)	11–77	N/A
Self-group overlap (adapted)	1–7	N/A

^aWEMWBS mean provided is for age group 18–29 years.

Finally, we were able to link participants' survey data with their usage data in the MindMax database. This enabled us to quantify their usage of MindMax, which we measured with the total number of seconds MindMax was open on their smartphone or other smart device. This figure reports the total duration of all of a user's sessions recorded in the database, with each session having an error of ±1–29 s. More specific details on the error associated with MindMax session timings are reported in a previous publication (Vella et al., 2018).

2.6. Analysis

2.6.1. Extracting GHSQ factors through an exploratory factor analysis

The authors of the GHSQ have suggested multiple ways to score it and also encourage it to be customised to better fit a research study's needs (Deane and Wilson, 2007). While this increases the utility of the GHSQ, it has also led to inconsistencies in reported factor structures and varying reports of scale reliability and validity (Hammer and Spiker, 2018). For this reason, after re-coding ‘Other’ string responses to the appropriate item (e.g. a specification of ‘Grandad’ was coded back to the ‘Other relative’ item), we ran an exploratory factor analysis using Mplus version 8.2 (Muthén and Muthén 1998-2017), with an estimation method of diagonally weighted least squares (referred to as WLSMV in Mplus) and a direct oblimin rotation (delta = 0), on the polychoric correlation matrices on our sample's (n = 313) Day 1 GHSQ responses. Polychoric correlations were chosen as they are more resistant to bias than Pearson correlations (Garrido et al., 2013). We also excluded the ‘MindMax’ item from this factor analysis and subsequent MANOVAs for clarity of interpretation. As can be seen in Table 2, all included items had a factor loading of >0.4, and therefore none were removed.

Table 2.

Two-factor rotated^a loadings, correlations, and communalities of our GHSQ items (n = 313).

	Personal	Impersonal	Communality
Intimate partner	0.446b	−0.110	0.184
Friend	0.547b	−0.198b	0.280
Parent	0.667b	0.069	0.475
Other relative	0.543b	0.218b	0.406
Mental health professional	−0.005	0.735b	0.539
Phone helpline	0.041	0.789b	0.641
Doctor/general practitioner	0.074	0.788b	0.657
Someone online, who you don't know	−.140b	0.676b	0.425
Personal factor correlation	1
Impersonal factor correlation	.269b	1

^aEstimation method WLSMV, with direct oblimin rotation. Bold indicates a loading high enough to consider for factor inclusion.

^bp < .05.

Bartlett's test of sphericity was significant, and the two-factor model displayed excellent fit (CLI = 0.983, TLI = 0.963). The first factor, which we termed ‘personal help-seeking’, was comprised of the items ‘intimate partner’, ‘friend’, ‘parent’, and ‘other relative’. The second factor, which we termed ‘impersonal help-seeking’, was comprised of the items ‘mental health professional’, ‘phone helpline’, ‘doctor/general practitioner’, and ‘someone online, who you don't know personally’. We calculated the Cronbach's alpha statistic for both factors and while it was high for impersonal help-seeking (α = 0.800), it was lower for personal help-seeking (α = 0.581). Additionally, the post-extraction communality for ‘intimate partner’ was below the accepted standard of 0.200 (Child, 2006). For these reasons, while we analysed the GHSQ as two subscale scores, our findings should be interpreted with some reservation. Both subscales have a minimum score of ‘4’ and a maximum score of ‘28’.

2.6.2. Primary analyses

Our primary analyses followed a modified intention-to-treat principle, whereby all possible casewise comparisons between Day 1 and Day 30, and Day 1 and Day 60, were performed, regardless of whether or not the participant had used MindMax (defined as logging at least one session lasting >0 s). This method was chosen because as all items within a survey (except demographics) were compulsory to answer, missing data would follow a file-matching pattern (all data would either be absent or present at each time point). Hence, two separate comparisons would maximise sample size, guard against overly small cell sizes, and ensure ease of analysis and interpretation. Imputation of missing data was not performed as previous research has found that imputing between-wave missing data results in no improvement, or even an increase, in standard error (Young and Johnson, 2015).

Analyses consisted of two 2 (time) × 2 (gender) × 2 (base wellbeing) mixed design MANOVAs assessing the impact of time (MANOVA 1: Day 1 vs. Day 30; MANOVA 2: Day 1 vs. Day 60), gender, and base wellbeing on participants' flourishing, resilience, personal help-seeking intentions, impersonal help-seeking intentions, and sense of connection to the MindMax community. Of our between-subjects factors, gender was split between males and females, and base wellbeing was split by the median of our sample's (n = 313) Day 1 WEMWBS score (median = 49).

2.6.3. Assumptions testing

We tested the assumptions of the mixed methods MANOVA for both comparisons (MANOVA 1 and MANOVA 2). For both MANOVAs, homogeneity of variance (Levene's test) and of variance-covariance matrices (Box's test) were violated for multiple cells, but the MANOVA is noted to be robust against such violations (O'Brien and Kaiser, 1985). Due to small cell sizes, we removed participants who were neither male nor female (n = 6 for MANOVA 1, and n = 7 for MANOVA 2) from the analysis.

Each of the dependent variable distributions in each cell of MANOVAs 1 and 2 was checked for normality. As the MANOVA is relatively robust to mild violations of normality (O'Brien and Kaiser, 1985), our criteria were kurtosis and skewness z-scores of z less than seven. Normality was severely violated for flourishing and self-group overlap, necessitating an inverse square root transformation for the former and a logarithmic transformation for the latter.

Each cell was also checked for univariate outliers (n = 13 for MANOVA 1, and n = 11 for MANOVA 2), and the subsamples were checked for multivariate outliers. We identified six multivariate outliers in the MANOVA 1 subsample (all except one case also identified as univariate outliers), and six multivariate outliers in the MANOVA 2 subsample (all except three cases also identified as univariate outliers). Hence, in total, 14 univariate and multivariate outlier cases were identified for MANOVA 1 and 14 univariate and multivariate outlier cases were identified for MANOVA 2. The outlying cases of both groups overlapped but were not fully identical.

To assess the impact of transformation and outliers on our findings, we conducted three sensitivity analyses: transformed; excluding outliers; and transformed excluding outliers.

All other assumptions of the mixed-methods MANOVA were satisfied. Table 3 shows correlations between the dependent variables, organised by time point.

Table 3.

Correlations between dependent variables.

	Dependent variable	Flourishing	CD-RISC 10	GHSQ (personal)	GHSQ (impersonal)	Self-group overlap
Day 1	Flourishing	1
	CD-RISC 10	0.616b	1
	GHSQ (personal)	0.360b	.311b	1
	GHSQ (impersonal)	0.093	0.080	0.223b	1
	Self-group overlap	0.103	0.105	0.075	0.137a	1
Day 30	Flourishing	1
	CD-RISC 10	0.698b	1
	GHSQ (personal)	0.450b	0.302b	1
	GHSQ (impersonal)	0.113	0.087	0.310b	1
	Self-group overlap	0.204a	0.184a	0.316b	0.417b	1
Day 60	Flourishing	1
	CD-RISC 10	0.616b	1
	GHSQ (personal)	0.496b	.357b	1
	GHSQ (impersonal)	0.078	0.061	0.347b	1
	Self-group overlap	0.119	0.191a	0.267b	0.477b	1

^ap < .05.

^bp < .001.

3. Results

3.1. Participant characteristics

Fig. 1 shows participant flow from recruitment to analysis.

Fig. 1.

Participant flow.

Table 4 displays sample age, base wellbeing, and app usage data, including range, means, medians, and SDs; while Table 5 displays other demographic frequency information. The subsamples analysed in MANOVA 1 and MANOVA 2 did not display a significant difference in demographics compared to the main sample. Non-completers of at least one follow-up survey (Day 30 or Day 60) also displayed no difference in demographics compared to the main sample apart from being more likely to have ever played video games (95.4% vs. 85.6%). For readability, we collapsed the following categories in Table 5 under the label ‘Not in work’: ‘Unemployed/looking for work’, ‘Home duties’, ‘Not working and currently receiving sickness allowance/disability support pension’, and ‘Volunteer work’. Similarly, an endorsement of ‘AFL involvement’ covers participant identification to any of these labels: ‘Amateur player’, ‘Administration’, ‘Supportive other’, or ‘Fan’.

Table 4.

Sample demographic means, medians, and SDs (n = 313).

Characteristic	Range	Mean (SD)	Median
Age (years)	16–35	23.4 (5.38)	23.0
Day 1 Wellbeing (WEMWBS)	19–70	47.9 (10.1)	49.0
Total usage (min)a	0–638	18.6 (47.3)	5.37

^a±1–29 s. 6 participants did not generate usage data.

Table 5.

Other self-reported sample demographic frequencies (n = 313).

Characteristic		n (%)a
Gender	Male	131 (41.9%)
	Female	174 (55.6%)
	Other	8 (2.60%)
Highest level of education	Primary school	1 (0.300%)
	Junior high school	7 (2.20%)
	Senior high school	117 (37.4%)
	Certificate/diploma	70 (22.4%)
	Degree	94 (30.0%)
	Postgraduate degree	24 (7.70%)
Ethnic background most identified with	English	48 (15.3%)
	Irish	10 (3.2%)
	Scottish	2 (0.6%)
	Italian	7 (2.2%)
	German	2 (0.60%)
	Chinese	28 (8.9%)
	Australian	138 (44.1%)
	Other	69 (22.0%)
Aboriginal and/or Torres Strait Islander	Yes	2 (0.60%)
	No	307 (98.1%)
Current main activities	Full-time work	92 (29.4%)
	Part-time work	39 (12.5%)
	School student	41 (13.1%)
	Tertiary education	92 (29.4%)
	Not in work	47 (14.6%)
Ever played video games	Yes	268 (85.6%)
	No	45 (14.4%)
AFL involvement	Yes	145 (46.6%)
	No	166 (53.0%)

^aPercentages do not sum to 100 due to rounding and also as participants could also state that they did not know or choose not to answer.

3.2. Missing data

As shown in Fig. 1, a considerable amount of dropout and protocol deviation (whether by not downloading MindMax, or by not answering follow-up surveys) was observed after Day 1. As those who only completed Day 1 (241/554, 43.5%) did not differ significantly in demographics to those who completed Day 1 and at least one follow-up survey (313/554, 56.5%), we judged it more appropriate to drop them from analysis instead of imputing large amounts of data.

Little's MCAR test was run on the remaining sample (n = 313) at both item and scale levels. Neither test returned a significant result. Hence, the data of this group was assumed to be missing completely at random and we proceeded to apply complete case analysis (in the form of 2 separate MANOVAs).

3.3. Analysis

3.3.1. Overall

Table 6, Table 7 present the (untransformed) subsample means and SDs for all outcome variables in MANOVAs 1 and 2, split by the between-subjects factors (gender and base wellbeing).

Table 6.

Dependent variable observed means (SD) for MANOVA 1 (Day 1 to Day 30).

Dependent variable		Total (n = 290)	Male (n = 124)	Female (n = 166)	Low WBa (n = 141)	High WBa (n = 149)
Flourishing	Day 1	41.9 (7.16)	42.4 (7.21)	41.6 (7.12)	37.9 (7.01)	45.8 (4.84)
	Day 30	42.1 (7.74)	42.6 (7.41)	41.8 (7.98)	39.1 (7.70)	45.1 (6.58)
CD-RISC 10	Day 1	26.1 (7.16)	27.4 (6.82)	25.2 (7.28)	22.6 (7.36)	29.5 (5.10)
	Day 30	26.2 (6.84)	27.7 (6.33)	25.0 (7.00)	23.2 (7.03)	29.0 (5.35)
GHSQ (personal)	Day 1	18.1 (4.70)	17.7 (4.53)	18.4 (4.81)	16.3 (5.03)	19.8 (3.61)
	Day 30	18.2 (4.51)	18.3 (4.66)	18.2 (4.41)	16.7 (4.88)	19.6 (3.62)
GHSQ (impersonal)	Day 1	13.7 (6.09)	15.0 (6.87)	12.7 (5.23)	12.1 (4.99)	15.1 (6.67)
	Day 30	14.2 (6.08)	15.7 (6.81)	13.1 (5.22)	12.9 (5.50)	15.5 (6.35)
Self-group overlap	Day 1	1.80 (1.31)	1.96 (1.38)	1.69 (1.25)	1.68 (1.40)	1.92 (1.22)
	Day 30	2.56 (1.57)	3.01 (1.67)	2.23 (1.40)	2.12 (1.38)	2.98 (1.62)

^aWB = wellbeing.

Table 7.

Dependent variable observed means (SD) MANOVA 2 (Day 1 to Day 60).

Dependent variable		Total (n = 239)	Male (n = 100)	Female (n = 139)	Low WBa (n = 119)	High WBa (n = 120)
Flourishing	Day 1	42.3 (6.81)	42.9 (6.42)	41.8 (7.06)	38.8 (6.72)	45.8 (4.85)
	Day 60	43.3 (6.91)	43.6 (6.88)	43.1 (6.95)	40.5 (7.00)	46.1 (5.57)
CD-RISC 10	Day 1	26.2 (6.79)	27.6 (6.14)	25.2 (7.08)	23.2 (6.93)	29.2 (5.16)
	Day 60	26.2 (6.07)	27.5 (5.69)	25.3 (6.20)	24.0 (6.38)	28.4 (4.85)
GHSQ (personal)	Day 1	18.0 (4.63)	17.8 (4.45)	18.2 (4.76)	16.2 (4.92)	19.8 (3.51)
	Day 60	18.3 (4.72)	17.6 (5.03)	18.7 (4.44)	16.7 (5.21)	19.8 (3.59)
GHSQ (impersonal)	Day 1	13.3 (5.78)	14.6 (6.57)	12.4 (4.94)	11.8 (4.86)	14.8 (6.24)
	Day 60	14.5 (5.43)	15.5 (5.62)	13.7 (5.18)	13.1 (4.96)	15.8 (5.58)
Self-group overlap	Day 1	1.92 (1.41)	2.17 (1.44)	1.75 (1.36)	1.81 (1.54)	2.04 (1.27)
	Day 60	2.79 (1.60)	3.24 (1.62)	2.46 (1.52)	2.26 (1.39)	3.31 (1.63)

^aWB = wellbeing.

Table 8 presents the multivariate effects of time, and higher-order interaction effects between time and other between-subjects factors for MANOVA 1 (comparing Day 1 to Day 30) and MANOVA 2 (comparing Day 1 to Day 60). We present partial eta squared values as measures of effect size, to be interpreted according to Cohen's recommended scales of magnitude (partial η² = 0.010, 0.059, and 0.138 for small, medium, and large effects respectively) (Cohen, 1988; Richardson, 2011). Pillai's Trace is an appropriate statistic to report due to the violation of homogeneity of variance-covariance matrices; however, as it is identical to partial eta squared in a repeated-measures MANOVA, we do not report it additionally. Finally, as two MANOVAs were conducted all p-values were assessed to a Bonferroni-corrected significance value of p = .025.

Table 8.

Multivariate and univariate effects of time and interaction effects for MANOVAs 1 and 2. Bold rows denote multivariate effects.

	MANOVA 1 (Day 1–30; n = 290)			MANOVA 2 (Day 1–60; n = 239)
	Df	Partial η2	F value	Df	Partial η2	F value
Time	5, 282	0.177	12.117a	5, 231	0.246	15.048a
Flourishing	1, 286	0.002	0.600	1, 235	0.022	5.402a
CD-RISC 10	1, 286	0.000	0.141	1, 235	0.000	0.056
GHSQ (personal)	1, 286	0.001	0.341	1, 235	0.003	0.696
GHSQ (impersonal)	1, 286	0.019	5.478a	1, 235	0.062	15.491a
Self-group overlap	1, 286	0.169	58.260a	1, 235	0.201	59.043a
Time × Gender	5, 282	0.025	1.421	5, 231	0.010	0.453
Flourishing	1, 286	0.001	0.358	1, 235	0.001	0.132
CD-RISC 10	1, 286	0.004	1.282	1, 235	0.000	0.032
GHSQ (personal)	1, 286	0.012	3.336	1, 235	0.005	1.171
GHSQ (impersonal)	1, 286	0.002	0.482	1, 235	0.001	0.128
Self-group overlap	1, 286	0.012	3.370	1, 235	0.003	0.674
Time × base wellbeing	5, 282	0.072	4.386a	5, 231	0.096	4.894a
Flourishing	1, 286	0.021	6.097a	1, 235	0.010	2.352
CD-RISC 10	1, 286	0.012	3.406	1, 235	0.023	5.621a
GHSQ (personal)	1, 286	0.008	2.238	1, 235	0.004	0.832
GHSQ (impersonal)	1, 286	0.004	1.068	1, 235	0.004	0.859
Self-group overlap	1, 286	0.038	11.248a	1, 235	0.056	13.817a
Time × gender × base wellbeing	5, 282	0.047	2.803a	5, 231	0.053	2.567
Flourishing	1, 286	0.001	0.422	1, 235	0.001	0.183
CD-RISC 10	1, 286	0.000	0.068	1, 235	0.000	0.054
GHSQ (personal)	1, 286	0.001	0.178	1, 235	0.004	1.049
GHSQ (impersonal)	1, 286	0.009	2.511	1, 235	0.041	10.040a, b
Self-group overlap	1, 286	0.029	8.397a	1, 235	0.006	1.473

^ap < .025.

^bNot followed up on due to multivariate non-significance.

3.3.2. MANOVA 1 (Day 1 vs. Day 30)

3.3.2.1. 30-day change in impersonal help-seeking

Alongside the observed impersonal help-seeking means on Day 1 and Day 30 (Table 6), the significant univariate main effect of time on impersonal help-seeking in MANOVA 1 (Table 8) indicates an increase in participants' impersonal help-seeking intentions from Day 1 to Day 30. Pairwise comparisons found an increase of 0.720 (95% CI = 0.114–1.33) in impersonal help-seeking estimated marginal means, p = .020.

3.3.2.2. Effect of base wellbeing on 30-day change in flourishing

Fig. 2 shows flourishing scores on Day 1 and Day 30, split by base wellbeing. Alongside the observed means of flourishing split by base wellbeing (Table 6) and the univariate significance of the time × base wellbeing interaction effect on flourishing in MANOVA 1 (Table 8), it suggests an increase in low-wellbeing participants but a decrease in high-wellbeing participants. However, pairwise comparisons found a nonsignificant change in both the low-wellbeing group and the high-wellbeing group, p > .025, suggesting that this pattern of results is due to chance.

Fig. 2.

Estimated marginal mean flourishing scores across time, split by base wellbeing. Error bars show standard error (SE). Asterisks indicate significant differences across time. (None present in Fig. 2.)

3.3.2.3. Effects of gender and base wellbeing on 30-day change in self-group overlap

Alongside the observed self-group overlap means on Day 1 and Day 30 (Table 6), the significant univariate main effect of time on self-group overlap in MANOVA 1 (Table 8) indicates an increase in participants' sense of connection to the MindMax community from Day 1 to Day 30. However, the significant interaction effects of time × base wellbeing and time × gender × base wellbeing on self-group overlap in MANOVA 1 suggest that this increase may be driven by certain groups. We followed up the time × gender × base wellbeing interaction to investigate this further.

Fig. 3 shows self-group overlap scores on Day 1 and Day 30, split by gender and base wellbeing. It suggests that self-group overlap increased more in high-wellbeing males compared to low-wellbeing males and females in general. To test this, we split the sample by gender and conducted mixed design MANOVAs on each subsample. The time × base wellbeing interaction had a medium-to-large, significant effect on self-group overlap scores in male participants, F(1,122) = 12.1, p = .001, partial η² = 0.090, and a negligible, nonsignificant effect in female participants, F(1,160) = 0.962, p > .025, partial η² = 0.029. This suggests that base wellbeing only influenced the change in self-group overlap in our male participants. Pairwise comparisons found an increase of 1.49 (95% CI = 1.14–1.83) in self-group overlap estimated marginal means in high-wellbeing males, p < .001, and an increase of 0.505 (95% CI = 0.191–0.819) and 0.592 (95% CI = 0.228–0.955) in low-wellbeing and high-wellbeing females respectively, both p = .002. The change in self-group overlap in low-wellbeing males was nonsignificant, p > .025, which suggests it is due to chance.

Fig. 3.

Estimated marginal mean self-group overlap scores across time, split by gender and base wellbeing. Error bars show SE. Asterisks indicate significant differences across time.

3.3.3. MANOVA 2 (Day 1 vs. Day 60)

3.3.3.1. 60-day change in flourishing

Alongside the observed flourishing means on Day 1 and Day 60 (Table 7), the significant univariate main effect of time on flourishing in MANOVA 2 (Table 8) indicates an increase in participants' flourishing from Day 1 to Day 60. Pairwise comparisons found an increase of 1.01 (95% CI = 0.154–1.87) in flourishing estimated marginal means, p = .021.

3.3.3.2. 60-day change in impersonal help-seeking

Alongside the observed impersonal help-seeking means on Day 1 and Day 60 (Table 7), the significant univariate main effect of time on impersonal help-seeking in MANOVA 2 (Table 8) indicates a non-chance increase in participants' impersonal help-seeking intentions from Day 1 to Day 60. Pairwise comparisons found an increase of 1.35 (95% CI = 0.674–2.03) in impersonal help-seeking estimated marginal means, p < .001.

3.3.3.3. Effect of base wellbeing on 60-day change in self-group overlap

Alongside the observed self-group overlap means on Day 1 and Day 60 (Table 7), the significant univariate main effect of time on self-group overlap in MANOVA 2 (Table 8) indicates a non-chance increase in participants' sense of connection to the MindMax community from Day 1 to Day 60. However, the significant interaction effect of time × base wellbeing on self-group overlap suggests that this increase may be influenced by level of base wellbeing. We followed up the time × base wellbeing interaction to investigate this further.

Fig. 4 shows self-group overlap scores on Day 1 and Day 60, split by base wellbeing. Pairwise comparisons found an increase of 0.439 (95% CI = 0.119–0.757) in self-group overlap estimated marginal means in low-wellbeing participants, p = .007, and an increase of 1.26 (95% CI = 0.963–1.56) in high-wellbeing participants, p < .001. The significant interaction suggests that the higher increase in high-wellbeing participants is not due to chance.

Fig. 4.

Estimated marginal mean self-group overlap scores across time, split by base wellbeing. Error bars show SEs. Asterisks indicate significant differences across time.

3.3.3.4. Effect of base wellbeing on 60-day change in resilience

Finally, Fig. 5 shows resilience scores on Day 1 and Day 60, split by base wellbeing. Alongside the observed means of resilience split by base wellbeing on Day 1 and Day 60 (Table 7) and the univariate significance of the time × base wellbeing interaction effect on resilience in MANOVA 2 (Table 8), it suggests an increase in low-wellbeing participants but a decrease in high-wellbeing participants. However, pairwise comparisons found a nonsignificant change in both the low-wellbeing group and the high-wellbeing group, p > .025, suggesting that this pattern of results is due to chance.

Fig. 5.

Estimated marginal mean resilience scores across time, split by base wellbeing. Error bars show SEs. Asterisks indicate significant differences across time. (None present in Fig. 5.)

3.4. Sensitivity analyses

Sensitivity analyses were performed to assess the impact of transforming our sample's flourishing and self-group overlap data, as well as the impact of removing univariate and multivariate outliers, for MANOVA 1 and MANOVA 2. All significant main effects of time identified in the primary analysis were also present in the sensitivity analyses.

Transformation of our skewed variables (flourishing and self-group overlap) did not lead to any changes in statistical significance for any of the group differences reported above. However, removal of outliers did. In MANOVA 1 (change between Days 1–30), the time × base wellbeing and time × gender × base wellbeing interaction effects on self-group overlap were nonsignificant when outliers were removed, and a significant time × gender interaction effect was detected at a multivariate level, F(5, 266) = 3.208, p = .008, partial η² = 0.057, and at a univariate level for self-group overlap, F(1, 270) = 11.781, p = .001, partial η² = 0.042. Pairwise comparisons found an increase of 1.19 (95% CI = 0.919–1.46) in self-group overlap estimated marginal means in male participants, p < .001, and an increase of 0.580 (95% CI = 0.359–0.801) in female participants, p < .001.

In MANOVA 2 (change between Days 1–60), the univariate time × base wellbeing interaction effect on self-group overlap was no longer present once a logarithmic transformation was applied and outliers were removed (but it was present in both the transformed-only analysis and the outliers-removed analysis).

4. Discussion

4.1. Primary result

In this naturalistic study, we provided participants with access to MindMax, which they used as they wished. The aim of this study was to identify whether we would observe a change in participants' flourishing, resilience, help-seeking intentions, and sense of connection to the MindMax community across time, and whether gender and base wellbeing influenced any such change. Two mixed design MANOVAs were conducted to assess 30-day change from baseline and 60-day change from baseline. We identified 30-day increases in impersonal help-seeking intentions and sense of connection to the MindMax community, and 60-day increases in flourishing, impersonal help-seeking intentions, and sense of connection to the MindMax community. While the 30-day increase in impersonal help-seeking intentions and 60-day increase in flourishing and impersonal help-seeking intentions were observed across the sample, both the 30-day and 60-day increases in sense of connection were influenced by either one or both of our between-subjects factors (gender and base wellbeing).

Specifically, the 30-day increase in sense of connection was highest in our male participants with high (above median) base wellbeing. A smaller increase was also observed in our female participants regardless of their level of base wellbeing. We observed no increase in male participants with low base wellbeing.

Gender did not influence 60-day increases in sense of connection to the MindMax community; however, we observed a smaller increase in low-wellbeing participants than in high-wellbeing participants.

The various interaction effects involving time, gender, and base wellbeing on self-group overlap were not robust against corrections for extreme positive skew, and could therefore also be attributed to chance, unique participant circumstances (for the outlying cases), or a lack of statistical power. However, the 30-day and 60-day increases in impersonal help-seeking intentions and sense of connection to the MindMax community, and the 60-day increase in flourishing remained robust. We can hence confidently state that these constructs increased over the 30-day and 60-day intervals in which our participants were given the opportunity to use MindMax.

4.2. Help-seeking intentions

In this study, participants' personal help-seeking scores were the sum of their willingness to seek help for personal-emotional problems from their ‘intimate partner’, ‘friend’, ‘parent’, or ‘other relative’, while their impersonal help-seeking scores were the sum of their willingness to seek help for personal-emotional problems from their ‘doctor/general practitioner’, ‘mental health professional’, ‘phone helpline’, and ‘someone online, who you don't know personally’ (Table 2). As shown in Table 6, participants' personal help-seeking scores trended higher than their impersonal help-seeking scores (although this difference was not assessed statistically). In conjunction with this, male participants showed a trend of having higher impersonal help-seeking scores than female participants, while the pattern was opposite for personal help-seeking scores. Finally, consistent with previous research (Rickwood et al., 2005), low-wellbeing participants had lower help-seeking scores than high-wellbeing participants.

We observed an overall increase in impersonal help-seeking intentions over 30 days and 60 days from baseline, regardless of gender and base wellbeing (Table 6; Table 7). MindMax's target audience is younger men interested in AFL and/or video games, and MindMax features young male role models and uses a casual masculine tone. It addresses the lack of mental health apps aimed specifically at a younger male audience (Rice et al., 2018). As Australian men exhibit low rates of help-seeking despite their level of need (Rice et al., 2018), our observed increase in impersonal help-seeking among our male participants is encouraging. Furthermore, our results suggest that this gender-specific targeting did not seem to have had any adverse effect on our female participants' intentions to seek help from informal sources.

While an exploratory factor analysis conducted on our GHSQ items (Table 2) provided support for grouping these sources together, it must be noted that the overall solution was not ideal due to lower-than-desired commonality of the item ‘intimate partner’ and internal consistency of the items making up the personal help-seeking factor. However, there were no such problems with the impersonal help-seeking factor and its associated items. With these qualifications in mind, the grouping of ‘someone online’ with the other, more formal sources (Ellis et al., 2013) in the same factor suggests that people who endorse or reject these items may not do so due to their perceived authority, or clinical or service-related nature, but rather their distance from their personal sphere. This is not surprising when viewed in the context of research on the links between threatened masculinity, shame, and how these interact with mental health stigma to discourage help-seeking (Rice et al., 2018), and may be indicative of a shift in help-seeking patterns.

4.3. Sense of connection to the MindMax community

We observed an overall increase in sense of connection to the MindMax community (self-group overlap) over 30 days and 60 days from baseline (Table 6, Table 7, Table 8). While this increase appeared to be driven by gender and base wellbeing (with the largest increases observed in our male participants and our participants with high base wellbeing), these interaction effects were not robust to sensitivity analyses excluding outliers, and could therefore be due to exceptional cases. Regardless, the overall increase we observed is encouraging, as it suggests that MindMax was successful in creating a community of users.

Our primary analysis detected no significant 30-day increase in sense of connection to the MindMax community in our male participants with low base wellbeing (Fig. 3). A possible explanation for this may be the tendency of low-wellbeing individuals to feel more isolated and withdraw from prosocial activity (Deci and Ryan, 2000), compounded with the tendency for men to have lower levels of emotional competence than women in general (Ellis et al., 2014). However, the interaction effect between time, gender, and base wellbeing on self-group overlap was not present when assessing 60-day change in sense of connection to the MindMax community. This indicates that being male did not adversely influence this increase, and suggests that men with lower wellbeing may take longer to establish a meaningful sense of connection. Alternatively, as assessments of 30-day change and 60-day change only included participants who answered surveys at those time points, it may be reflective of the characteristics of the subgroups. We note, however, that demographic characteristics did not differ across subgroups (Section 3.1).

It is also encouraging that our female participants endorsed an increased sense of connection to the MindMax community over time, despite female participants in an earlier study expressing a lack of identification with its masculine tone (Cheng et al., 2018). While MindMax is not a perfect example of a gender-synchronised intervention as recommended by Rice and colleagues (Rice et al., 2018), this finding suggests that it at least does not actively create barriers for its female users.

4.4. Flourishing

Finally, we observed an overall increase in flourishing over 60 days from baseline (Table 7; Table 8). However, no change was observed 30 days from baseline. This suggests that a longer amount of time was required to achieve significant increases in flourishing across all subgroups (split by gender or base wellbeing). An alternative explanation is that the same background factors that influenced individuals into continued participation in the trial (by answering surveys 60 days after baseline) could also have contributed to this increase. Regardless, it is again an encouraging observation that may point to MindMax's usefulness in a grander strategy for improving mental health and wellbeing.

4.5. Limitations and future research

There are several limitations to this study that must be considered alongside the above reported findings. First, this trial was naturalistic by design, with one condition. As a result, these findings are observational and we cannot make claims about the impact of the absence or presence of participants' exposure to MindMax. While the 30-day and 60-day increases in impersonal help-seeking intentions, sense of connection to the MindMax community, and flourishing could be due to exposure to MindMax and its content, they could also be due to an underlying factor that could have influenced both participants' enrolment in this study and their subsequently observed changes. The single-arm design also meant that we were unable to test MindMax's intervention principles against other principles or even a control condition, and that MindMax's efficacy and effectiveness would have to be fully tested through a randomised controlled trial.

However, a strength of this design is that it accommodated natural usage patterns, leading to increased external validity, and further allowed us to observe participants' usage. Our results are hence able to be interpreted in the context of app usage (although small cell sizes prevent it from being used as a predictor variable). We have published research focusing more specifically on participant usage of, and engagement with, MindMax (Vella et al., 2018), and plan to explore usage and wellbeing more fully in future publications.

The use of median splits to form analysis groups leads to an increased risk of regression to the mean. Certain patterns of change we observed over time (Fig. 2; Fig. 5) are consistent with, and may be explained by, this pattern.

We also observed a considerable amount of dropout (though at a rate characteristic of eHealth evaluations; Fleming et al., 2018), particularly after Day 1. Hence, full implementation of intention-to-treat analysis principles was not possible, particularly as previous research has found that imputing entire waves of data has no real impact on analysis efficiency (Young and Johnson, 2015). Therefore, the results of this study cannot be generalised outside the context of participants who had been given the opportunity to use MindMax and who had answered the surveys at the time points specified. Missing data was also observed within the analysis sample (n = 313), though Little's test suggests that it was missing completely at random. However, sensitivity analyses assuming a less random type of missingness (e.g. missing at random) could not be performed for the same reason stated above. Furthermore, while we assumed the data to be missing completely at random, there is still the possibility that the two analyses (MANOVA 1 and MANOVA 2) could have been performed on systematically different groups.

MindMax also changed across the duration of the trial, with multiple functional and aesthetic updates applied to the app (described in Section 2.2). While these changes could have adversely affected users (including participants of this study), all these changes intended to improve MindMax's content delivery, and the delivery of the intervention principles through which it aimed to effect wellbeing, resilience, help-seeking intentions, and sense of connection to the MindMax community. Furthermore, as constant updates to content and delivery are actually the norm in the wider, non-academic delivery of electronic services, this resulted not only in greater ecological validity, but also increased MindMax's ability to remain relevant to its users (Fleming et al., 2016; Mohr et al., 2015). Of course, it is also possible that the varying versions of MindMax throughout the trial each had slightly different effects, which may have affected our results. It must also be noted that the current version of MindMax is not the same version that was evaluated in the research reported in this paper.

Of note is the demographic distribution of the naturalistic trial sample. Just over half the sample was female (Table 5), which may reflect the higher level of interest women tend to have in eHealth programmes (Batterham and Calear, 2017) and in participating in research trials (Diviak et al., 2006). Additionally, less of the participants were involved with AFL (whether as players, staff, or fans) than compared to the organic (i.e. non-trial, as MindMax was available to the public during the trial) user base, which may have been due to the modality of trial recruitment (i.e. online advertising, and paper advertising in the Australian states of Queensland and New South Wales, where AFL is not as popular). Mentioning vouchers in recruitment materials may also have influenced decisions to participate in, and therefore the results of, this study.

The naturalistic trial sample differs demographically from MindMax's organic user base, which expressed a high mean level of interest in AFL and was over 80% male (Vella et al., 2018). Again, this suggests that in general (beyond the context of an evaluation trial), MindMax succeeded at appealing to its target population. Furthermore, given that some increases we observed in our trial participants were higher in males (e.g. sense of connection to the MindMax community), these effects are not likely to be diminished, and may even be magnified, in MindMax's organic user base. While most observed effect sizes were small, in population-wide initiatives that apps like MindMax have the potential of becoming, small effects may result in considerable aggregate gains.

Finally, as this study focused primarily on wellbeing, we were unable to obtain data on our participants' conformity to masculine norms, particularly harmful ones (The Men's Project and Flood, 2018). Future research can consider investigating how this may influence mHealth engagement, and how these may affect help-seeking behaviours.

4.6. Conclusion

This article reports the changes we observed in participants who were given the opportunity to use MindMax, an AFL-themed app that combines psychoeducation, social connection, and applied games. We asked participants to use MindMax at their leisure and assessed their 30-day and 60-day change from baseline in their wellbeing, resilience, and help-seeking scores. As recommended by its developers, we customised the GHSQ (Deane and Wilson, 2007; Wilson et al., 2005), and an exploratory factor analysis revealed two factors: personal and impersonal help-seeking. We observed 30-day increases in impersonal help-seeking intentions and sense of connection to the MindMax community, and 60-day increases in impersonal help-seeking intentions, sense of connection to the MindMax community, and flourishing. MindMax attempts to address the lack of mental health initiatives aimed directly at younger men (Rice et al., 2018; Seidler et al., 2016), incorporating co-design throughout its development (Cheng et al., 2018). In addition to exploring the impact of such initiatives on male help-seeking, future initiatives targeting men could also benefit from directly investigating the role of conformity to masculine norms and how rigid adherence to harmful aspects of masculinity may impact the effect of mental health and wellbeing interventions on an individual.

Funding

This work was supported by the Movember Foundation.

Declaration of competing interest

Professor Ian Hickie was an inaugural Commissioner on Australia's National Mental Health Commission (2012–18). He is the Co-Director, Health and Policy at the Brain and Mind Centre (BMC), University of Sydney. The BMC operates an early-intervention youth services at Camperdown under contract to headspace. Professor Hickie has previously led community-based and pharmaceutical industry-supported (Wyeth, Eli Lily, Servier, Pfizer, AstraZeneca) projects focused on the identification and better management of anxiety and depression. He was a member of the Medical Advisory Panel for Medibank Private until October 2017, a Board Member of Psychosis Australia Trust, and a member of Veterans Mental Health Clinical Reference group. He is the Chief Scientific Advisor to, and an equity shareholder in, InnoWell. InnoWell has been formed by the University of Sydney and PwC to deliver the Aus $30M Australian Government-funded ‘Project Synergy’. Project Synergy is a three-year program for the transformation of mental health services through the use of innovative technologies. None of the other authors declare any conflicts of interest.