Beyond clicks and downloads: a call for a more comprehensive approach to measuring mobile-health app engagement

Abstract

Downloading a mobile health (m-health) app on your smartphone does not mean you will ever use it. Telling another person about an app does not mean you like it. Using an online intervention does not mean it has had an impact on your well-being. Yet we consistently rely on downloads, clicks, ‘likes’ and other usage and popularity metrics to measure m-health app engagement. Doing so misses the complexity of how people perceive and use m-health apps in everyday life to manage mental health conditions. This article questions commonly used behavioural metrics of engagement in mental health research and care, and proposes a more comprehensive approach to measuring in-app engagement.

There is no shortage of mobile health (m-health) apps for people facing mental health challenges; one 2018 study estimated 10 000 available for download.¹ m-health apps track symptoms, deliver therapy and promote health behaviour interventions. Their success is often evaluated according to user engagement, typically operationalised as frequency and duration of app use, behavioural interaction with the app (for example downloads, clicks) and popularity (for example user reviews, ratings).² Usage data is assumed to capture different types and depths of app engagement, yet misses cognitive and emotional responses to the app and is disconnected from behaviour change in real-world settings.^3,4

User engagement has become an umbrella term used to describe a host of conceptually unique user-centred outcomes, including usability, acceptability, feasibility and satisfaction.⁵ These different user experiences are all evaluated with the same kinds of usage statistics: dwell time, bounce rates and number of downloads, logins, visits and specific interactions, such as clicking on links, watching videos, completing modules.² This creates a schism between the concept of interest and the most salient metrics for its evaluation. It opens the door ‘for [user-engagement indicators] to be selected inappropriately, presented with bias or interpreted incorrectly’ and prevents meaningfully comparisons across apps, studies and user groups.⁵

m-health engagement is narrowly defined as ‘user uptake … and/or ongoing use, adherence, retention, or completion data’,⁶ and thus focuses on quantifying rather than qualifying user engagement. Although it is seemingly objective, easy and unobtrusive to record usage data, studies have documented inconsistencies and called for standardised reporting practices.^2,5,7 In addition, the positive impact of m-health apps on real-world or clinical trial outcomes is inconclusive,^2,7 and the ‘beneficial dose [of apps] … or amount of exposure’ at the population level is unknown.² This calls into question which metrics and what thresholds may be indicative of user engagement for different apps, users or mental health conditions.

Steady and sustained app use is typically viewed as positive, disengagement and non-use as negative. This emphasis on user behaviour in user-engagement evaluation misses important information about users and their contexts. User engagement may be influenced by how content is organised and presented, symptom burden, environmental stressors and supports, and the desire for social connectivity.⁸ These cognitive, emotional and social factors may be evaluated through surveys, interviews and app reviews,^5,8 sometimes independent of usage data. Consequently, data sources may be disconnected and unable to inform each other.

It has been proposed that the field of m-health evaluation could be advanced if we understood the relationship between out-of-app and in-app engagement.³ However, a barrier to this is the way in which in-app user engagement is currently conceptualised and measured.

(Re)Defining user engagement in the m-health space

Human–computer interaction (HCI) researchers define user engagement as capturing and maintaining the attention and interest of technology users,⁹ and the cognitive, emotional and temporal investment made by users.¹⁰ Mental health interventions target not only behaviour, but emotional and cognitive processes,¹¹ and social connectivity may be fundamental for self-management.⁸ A holistic definition of user engagement with m-health apps goes beyond what people do to how these tools address needs for information/education, social support and personal agency.

HCI researchers have identified attributes of user engagement, including user attention, interest, motivation, control and system usability and aesthetic appeal.^9,10 Focusing on user-engagement attributes offers a targeted approach to measurement. For example, we might use eye-tracking or heat maps to gauge in-app attention, or brief self-report instruments to capture users’ sense of control at defined points in time during an interaction. HCI researchers have modelled user engagement as having natural ebbs and flows over the course of users’ interactions with digital tools. The process of model of engagement suggests that users move through points of engagement, periods of sustained engagement, disengagement and re-engagement and that some attributes are more salient at particular stages than others.¹⁰ Process-based models reveal that engagement is not an ‘all or nothing’ phenomenon. Thus, is it essential to not only measure interaction outcomes (i.e. total session duration, total app downloads) but users’ journeys through an app, i.e., ups and downs in interactivity and varying levels of emotional and cognitive involvement.

Qualifying the quantification of m-health app usage

Interpreting the ‘ups and downs’ of in-app user engagement involves looking at usage data differently, and connecting multiple data sources in significant ways. There are ‘unique cognitive, neurological or motor needs arising from mental illness’.⁵ People with chronic health conditions have fluctuating needs, yet apps do not take into account the diversity of individual lived experiences or different users, for example, young people, those newly diagnosed.⁸ High usage is not indicative of positive clinical outcomes but may actually reflect worsening mental health. App usage may also exacerbate negative mood by providing poor quality/too much information, technical challenges⁸ or increasing user's awareness of distressing symptoms.¹²

Morton et al¹³ found that people living with bipolar disorder experienced both negative and positive emotions toward self-monitoring; the practice made them feel that they were managing their condition effectively compared with others, but was also ‘an unpleasant reminder that they were living with bipolar disorder’. Nuanced interpretation is not revealed through usage data alone.

Condition-specific knowledge is critical, and participatory design approaches are essential for gathering insights from people with lived experience and healthcare providers. Participatory design draws upon different methods (focus groups, ethnography) to co-create and co-evaluate prototypes with users throughout the design life cycle.¹⁴ Such methods are needed to identify what goals an app should fulfil, how people want to use it and for what purposes, and how design features can reflect user preferences and goals. Such knowledge can aid in the development of appropriate engagement indicators that will facilitate the interpretation of usage and other data sources.

Apps contain different types of content (for example educational articles, quizzes, symptom trackers, social connection tools). Looking at how frequently/long users interact with individual content pages or features may be less informative than grouping features according to function or the need they are intended to serve, such as education, symptom tracking, social support. It may be productive to distinguish content that people typically access once, for example quiz, versus multiple times, for example sleep or exercise logs, and to consider the sequence of content interactions in terms of scaffolding engagement. Rethinking the analysis of usage data would allow for richer interpretations of why people use apps in naturalistic settings, such as for routine maintenance, affirmation, social support. It could also help tailor content according to recovery or illness stage to reduce cognitive load.¹³

Algorithms facilitate the identification of engagement patterns based on interactions with apps over time, often based on duration (short versus long term) and frequency (low versus high). Such analyses reveal that different users have different use trajectories, but they do not explain why the pattern is occurring or its significance to clinical outcomes. Diverse and varied streams of data, including user self-reports, discussion forum transcripts, social network data and data from symptom severity, functioning or quality of life measures can be used to make sense of usage data. Cluster analysis is another option, whereby usage patterns are examined in concert with clinical, sociodemographic variables or other data sources, such as in-app text messages between users and coaches to identify reasons for app use/non-use.¹⁵

Users weigh the benefits of using m-health apps with factors such as usability, convenience, personal risk, for example privacy, and cognitive or emotional effort,¹³ which may not be accounted for in in-app assessments. Empirical studies utilising self-report data may not use validated self-report inventories or use them systematically.⁵ Thus, incorporating different data sources must balance the burden and risk of self-reporting with the benefits, emphasise replicability in the selection and use of self-report instruments, and transparency in data collection and reporting.

Toward a more comprehensive view of in-app engagement

In-app engagement must seek to support the cognitive, emotional and behavioural changes necessary for desired mental health outcomes, including symptom reduction, recovery and quality of life improvements. For this to occur, use/non-use must be connected to broader out-of-app goals, and the value of negative emotions, behaviours and cognitions relative to overall self-management must be considered. For example, non-use may be indicative of improved mental health and the need for less reliance on digital interventions.

It is tempting to want a magic, uniform formula for measuring user engagement, and this has been the appeal of usage data.

We argue instead that user-engagement indices for m-health apps must be:

• Corroborative, where different measures including usage data, symptom severity assessment scales and subjective outcome assessments (for example quality of life) are used to determine what meaningful engagement with the app entails.

• Outcome, rather than output, oriented. If m-health apps are meant to improve intervention effectiveness, then how they are used becomes more important than how often they are used.

• Process based, where we expect to see ebbs and flows in usage. Rather than labelling app users as low or high engagers based on algorithmic (non)-use, we should adopt participatory design approaches (for example journey mapping) to appreciate how different users interact with different features of the app over time.

• Expert driven, meaning that the expertise of people with lived experience and clinicians is included throughout the design process to identify salient needs (for example social support) and goals (for example establishing a routine, symptom management) and how these can be met with the app, as well as to inform aesthetic and content design choices.

User-engagement indices should be developed in parallel with the app itself, and draw upon condition-specific knowledge and multiple data sources. The COPE approach (Corroborative, Outcome oriented, Process based, Expert driven) necessitates collaboration among people with mental health conditions, healthcare providers and user-experience designers to develop m-health apps. Documenting each element of this framework would result in greater transparency about how design decisions were made, what is being measured and why, and how the resulting app fits into the broader mental health landscape.

Biographies

Heather O'Brien (pictured) is Associate Professor in the School of Information at University of British Columbia in Vancouver.

Emma Morton is an Institute of Mental Health Marshall Fellow in the Department of Psychiatry at the University of British Columbia.

Andre Kampen is a Doctoral student in the School of Information at the University of British Columbia.

Steven Barnes is a Senior Instructor in the Department of Psychology at the University of British Columbia.

Erin Michalak is a Professor in the Department of Psychiatry at the University of British Columbia.

Funding

This research is supported by a Canadian Institute for Health Research (CIHR) Project Grant, ‘Bipolar Bridges: A Digital Health Innovation Targeting Quality of Life in Bipolar Disorder’.

Author contributions

Conceptualised and original draft by H.L.O'B; conceptualised and reviewing: E.E.M. and E.M.; critical revision of the manuscript for important intellectual content: all authors.

Declaration of interest

H.L.O'B., E.M., A.K. and S.J.B.: none. E.E.M. discloses grant funding from a private company in the 36 months prior to this publication.