Systematic review of cost-effectiveness analysis of behavior change communication apps: Assessment of key methods

Abstract

Objective

Evidence backing the effectiveness of mobile health technology is growing, and behavior change communication applications (apps) are fast becoming a useful platform for behavioral health programs. However, data to support the cost-effectiveness of these interventions are limited. Suggestions for overcoming the low output of economic data include addressing the methodological challenges for conducting cost-effectiveness analysis of behavior change app programs. This study is a systematic review of cost-effectiveness analyses of behavior change communication apps and a documentation of the reported challenges for investigating their cost-effectiveness.

Materials and methods

Four academic databases: Medline (Ovid), CINAHL, EMBASE and Google Scholar, were searched. Eligibility criteria included original articles that use a cost-effectiveness evaluation method, published between 2008 and 2018, and in the English language.

Results

Out of the 60 potentially eligible studies, 6 used cost-effectiveness analysis method and met the inclusion criteria.

Conclusion

The evidence to support the cost-effectiveness of behavior change communication apps is insufficient, with all studies reporting significant study challenges for estimating program costs and outcomes. The main challenges included limited or lack of cost data, inappropriate cost measures, difficulty with identifying and quantifying app effectiveness, representing app effects as Quality-adjusted Life Years, and aggregating cost and effects into a single quantitative measure like Incremental Cost Effectiveness Ratio. These challenges highlight the need for comprehensive economic evaluation methods that balance app data quality issues with practical concerns. This would likely improve the usefulness of cost-effectiveness data for decisions on adoption, implementation, scalability, sustainability, and the benefits of broader healthcare investments.

Introduction

Mobile health (mHealth) applications (apps) designed for health promotion and disease management are growing in popularity. Currently, there are about 97,000 mHealth apps listed on 62 app stores. ¹ Coupled with the extensive market penetration of smartphones, mHealth apps have broadened in capacity, scope and reach. ² Apps that encourage healthy behaviors are classified among behavior change communication (BCC) programs. Recent evidence suggests that BCC programs are common and becoming increasingly successful. ³ Apps provide an ideal platform for BCC programs due to their ubiquity, connectivity and increased sophistication.^4,5 Compared to face-to-face programs, BCC apps have the potential to reach a larger number of people at more convenient times. ⁶ Common examples of BCC app programs include appointment reminders, support for medication adherence, community mobilization, and awareness campaigns. ⁷

Effectiveness studies on BCC apps point to effects, albeit small, in increasing access to health information, enhancing clinical services, and improving health behaviors.^8–13 Likewise, a systematic review showed that nearly 90% of BCC apps that targeted physical activity, weight loss, and mental health reported statistically significant behavior change. ⁸ Despite claims of success and potential for greater impact, evidence on the effectiveness of BCC apps needed for large-scale implementation is still sparse. ³ To date, few replication studies exist, and best strategies for effectiveness assessments and user engagement at scale are insufficient.^8,12,14,15 In part, institutional adoption is necessary to develop better evidence for program effectiveness, implementation and evaluation. ¹⁶

An additional impediment for the adoption and spread of BCC apps is the difficulty in estimating their economic value through cost-effectiveness analysis (CEA).^17–19 CEA studies compare the incremental costs and incremental effects of interventions with the goal of establishing the “economic value” of interventions, ²⁰ or alternatively, whether the incremental effectiveness justifies the additional cost of the apps. While BCC apps hold great promise, their place in behavioral health might depend on whether they are a cost-effective alternative for replacing or enhancing current practice. ²¹ The costs associated with the development and maintenance of app programs could enhance or impede their use on a broader scale. Furthermore, economic evaluation can play an important role in prioritizing health programs by enabling decision-makers to make better choices between interventions that provide the most cost-effective health outcomes.^22,23 As with the evidence on effectiveness, the economic evaluation literature on BCC apps is also sparse. De la Torre-Diez et al. indicated there were too few CEA studies in the literature. ¹⁹ Systematic reviews by Whitten et al. and Badawy et al. also reported a lack of adequate evidence of the extent to which app interventions represent a sensible priority for healthcare investment. Pointing to deficits in the quality of the economic evaluation, the authors also called attention to the shortfalls in study design methods for evaluating apps.^17,18

The objectives of this paper are to review cost-effectiveness analyses of BCC apps and to describe associated study challenges. Our goal is to inform public health researchers, health experts, and policymakers about the cost-effectiveness landscape of BCC apps, as well as highlight methodological challenges for conducting CEA studies on BCC apps.

Cost-effectiveness analysis

Cost-effectiveness studies compare the costs and outcomes of two or more health interventions. Generally, a new intervention is compared to “usual care” or a “do-nothing” alternative. ²⁴ The incremental cost of the new intervention is compared to the incremental outcomes to calculate the incremental cost-effectiveness ratio (ICER). Outcomes are measured as either “natural units,” such as cases averted or cases detected, or in Quality Adjusted Life Years (QALYs). QALYs is often preferred because life years adjusted for quality is a common outcome that can be used to compare disparate interventions. The QALY measure combines quality of life and survival outcomes into one effect, and is expressed by the sum of years lived weighting each year by a quality of life weight between 0 (dead) and 1 (perfect health). ²⁰ Hence, 1QALY equates to one year in perfect health. ICER is calculated by dividing incremental cost by incremental effect (usually in QALYs) to provide a ratio which represents the additional cost per additional unit of health effect. ²⁵ A lower ICER is preferred because it indicates that the incremental cost of an intervention is low per year of life gained adjusted for quality (QALY). The ICER from an intervention is compared to the ICER of other interventions or to a threshold that is used as a reference for value. In the US, by convention, interventions with an ICER below $100,000 per QALY ²⁶ are considered cost-effective, but acceptable thresholds range from $50,000 to $200,000 per QALY.

Finally, two important considerations are the analytical perspective and the time horizon. The perspective of a CEA refers to the viewpoint of the evaluation, which is based on the types of costs and health effects an analyst selects to estimate the ICER. The time horizon of a CEA is the follow-up period accounted for in the calculation of costs and outcomes used to compute the economic measures. Since QALYs take into account life expectancy, the time horizon is often long.

CEA follows the “Rule of Reason,” which states that the costs and effects expected to be trivially small with little impact on results, be reasonably excluded at the analyst’s discretion. ²⁷ While the “Rule of Reason” increases flexibility in conducting CEA, studies involving BCC apps are still challenging using recommended guidelines. A review of the CEA guidelines is therefore important to highlight methodological challenges that decrease the quality and usefulness of CEA evidence for BCC apps. Overcoming these challenges could largely expand the CEA evidence base for apps to make studies more comparable and supportive of healthcare decisions and resource allocation.

The panels’ recommendations

In 2016, the Second Panel on Cost-Effectiveness in Health and Medicine in the US (herein, the Second Panel) updated the First Panel’s CEA recommendations (APPENDIX C) with the same intent of study comparability. In summary, the Second Panel called on analysts to report costs and effects from societal and healthcare sector perspectives in addition to other analytic perspectives of interest. The Second Panel also recommended that an Impact Inventory be included in CEA reporting. An Impact Inventory is a catalogue of related and unrelated costs and effects of the intervention from societal and healthcare sector perspectives. The Impact Inventory aims to reduce barriers of low transparency and misunderstandings of CEA measures and how they were generated for cost-effectiveness calculations. ²⁶ Finally, it was recommended that CEA results be reported in Incremental Cost-Effectiveness Ratio (ICER), with health effects aggregated into a single measure of QALYs. ²⁰

Analytic perspective

The challenges faced by healthcare decision makers in separating CEA results from the costs and effects that are solely attributable to the healthcare sector are well documented.^28,29 In part, these challenges are due to the fact that many CEA studies report results from a societal perspective. ³⁰ A societal perspective measures the total cost and effect of an intervention regardless of who incurred them. Conversely, a healthcare sector perspective estimates the costs and effects of the intervention that is attributable to the healthcare sector without consideration for the costs and effects contributed by other sectors. ²⁶ The Second Panel’s recommendation for analysts to report results for both societal and healthcare sector perspectives is to increase the relevance and usefulness of CEA data for healthcare decision-making. ³¹

Methods

In this review, we: 1) selected relevant studies based on specific inclusion criteria, 2) developed and summarized study characteristics, and 3) assessed CEA studies using the Second Panel’s key recommendations for conducting CEA. ²⁶ We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2009 guidelines (Appendix B). ³²

Inclusion criteria

All cost-effectiveness analysis of BCC apps targeting a specific behavioral health outcome(s) were included in this review. The eligible studies were restricted to English language publications, with publication date ranging from January 1, 2008 to January 17, 2018.

Information source and search strategy

The search strategy was developed by the lead author and independently verified by a research librarian to ensure search comprehensiveness and accuracy of studies retrieved for the review. All discrepancies were discussed and agreed upon with the remaining authors. Information sources included Medline (Ovid), EMBASE, CINAHL (Cumulative Index to Nursing and Allied Health Literature) and Google Scholar. The databases were searched on January 17, 2018 to generate relevant studies using key search terms, such as “mHealth AND app* AND ((Cost* adj3 (Benefit* or Effectiveness or Utilit* or saving* or minimization))”. The full search strategy for Medline (Ovid) is shown in Table 1.

Table 1.

Search strategy in medline (ovid).

Number	Search	Results
Query string: The query string of subheadings (MeSH) and text words used for the database search were, “((Cost* adj3 (Benefit* or Effectiveness or Utilit* or saving* or minimization)) or (Economic adj3 (Evaluation* or analys*)) or (Marginal adj3 Analys*) or exp “Costs and Cost Analysis”/ AND [(((Mobile adj3 Health) or mHealth or m-Health or Telehealth or Tele-health or eHealth or e-Health) and (application or app or apps)) or (exp Telemedicine/ and exp Mobile Applications/) OR (((Mobile or Portable or electronic or software or cell or smartphone* or smart phone* or web-based) adj3 (application* or app or apps)) or mApps or m-Apps or m-App or mApp or m-Application* or mApplication* or exp Mobile Applications/]”.
1	(Cost* adj3 (Benefit* or Effectiveness or Utilit* or saving* or minimization)) or (Economic adj3 (Evaluation* or analys*)) or (Marginal adj3 Analys*) or exp "Costs and Cost Analysis"/	2,81,955
2	(((Mobile n3 Health) or mHealth or m Health or Telehealth or Tele health or eHealth or e Health) and (application or app or apps)) or (MH "Telehealth+" and MH "Mobile Applications")	2,807
3	((Mobile or Portable or electronic or software or cell or smartphone* or smart phone* or web-based) adj3 (application* or app or apps)) or mApps or m-Apps or m-App or mApp or m-Application* or mApplication* or exp Mobile Applications/	22,819
4	2 OR 3	23,727
5	1 AND 4	349
6	Remove duplicates from 5	304
7	Limit 6 to English language	297
8	Limit 7 to yr=”2008 – Current”	243

Study selection

The search results were exported into EndNote X8 citation manager, and duplicates were removed. The remaining citations were examined for relevant titles and abstracts. Systematic reviews, meta-analyses and non-original research studies were excluded. After the full text review, we further excluded studies that did not provide clear and sufficient information about CEA method or specific behavioral outcome(s) of interest. The references of eligible studies were scrutinized to find additional studies.

Data collection process

Microsoft Excel was used to extract study characteristics (see Appendix A): aim(s), intervention, participants, comparators, research perspective, time horizon, economic method used, outcome(s), outcome measure(s) and data source(s). When a study had both primary and secondary outcomes, the outcome directly related to the economic analysis was documented. Outcome was defined as any objective measure of behavioral outcome, that can be accurately captured by the BCC app.

Results

Study selection

The search strategy initially retrieved 694 studies: 243 in Medline, 80 in CINAHL, and 371 in Embase. Duplicates were removed, followed by studies without relevant titles and abstracts which resulted in 60 potentially eligible studies. The full text of the 60 potentially eligible studies were scrutinized, 51 of which were excluded. Nine studies were original research that used an economic evaluation method to assess a BCC app intervention. A search through the references of the nine studies identified 1 additional eligible study. Of these studies, 6 used a CEA method. Figure 1 shows the flow diagram for the final study selection.

Figure 1.

Flow chart showing final study selection.

Study characteristics

The 6 CEA studies were published between 2012 and 2017. Van Reijen M, et al. evaluated the “Strengthen your Ankle” mobile app, which provides written, visual, and verbal instructions as well as a calendar function, to help athletes prevent the recurrence of ankle sprains. ³³ Martín JAC, et al. evaluated “CardioManager”, a medical app which provides heart disease patients with the necessary behavioral and clinical guidelines to enable them to self-manage and monitor their own health conditions. ³⁴ The “Tät” app evaluated by Sjöström M, et al. was developed as a first line treatment for stress urinary incontinence based on self-management, which also provides patients with instructions for pelvic floor muscle training. ³⁵ Dahlberg K, et al. evaluated the “Recovery Assessment by Phone Points (RAPP)” app, which is a post-operative recovery monitoring program that allows patients to report progress of recovery directly to healthcare professionals after day surgery. ³⁶ The “Bring-your-own-device (BYOD)” mobile app evaluated by Armstrong K, at al. supports post-operative care in breast reconstruction patients by providing patients with validated quality of recovery questionnaires and surgical site photo submissions. Results from the questionnaire enables health professionals to detect post-operative complications and eliminate in-person follow-up care. ⁵ Ryan D, et al. evaluated the “t+ Asthma” app that enables twice daily transmission of symptoms, drug use, and peak flow, and prompts patients about agreed action plans. An incursion into the red or amber zones triggers a contact by an asthma nurse the next day. ²¹

The CEA studies were analyzed using the Second Panel’s recommendations and the study characteristics were described in Table 2. The BCC apps used in four studies (66.7%) were focused on patient adherence to behavioral and clinical guidelines for self-monitoring, while the apps in the remaining two studies required patient feedback to enable healthcare professionals conduct risk assessments. Two studies (33.3%) each reported a societal or a healthcare sector perspective. One study (16.7%) reported both a societal and healthcare sector perspective, and one study (16.7%) reported a health service provider perspective. The data sources for conducting the CEA varied across studies; four studies (66.7%) used questionnaires in addition to other primary data sources.

Table 2.

Characteristics of CEA studies.

CEA characteristics	Number of studies (%)
Focus of intervention
Patient adherence to behavioral and clinical guidelines for self-management	4 (66.7)
Provide feedback to healthcare professionals for risk assessment/reduction	2 (33.3)
Clearly stated reference case perspective
Yes	6 (100.0)
No	0 (0.0)
Reference case perspective
Societal	2 (33.3)
Healthcare Sector	2 (33.3)
Health Service Provider	1 (16.7)
Societal and Healthcare Sector	1 (16.7)
Data Source (quantifying cost and outcomes)
Questionnaire	4 (66.7)
Other	6 (100)
Discounted cost and/or outcomes
Yes	5 (83.3)
No	1 (16.7)
Outcome measure
Quality Adjusted Life Years (QALYs)	5 (83.3)
Other	1 (16.7)
Cost-effectiveness results reported as
Incremental cost-effectiveness ratio (ICER)	2 (33.3)
Other	4 (66.7)
Use of cost-effectiveness threshold
Yes	0 (0.0)
No	6 (100.0)
Sensitivity analysis
One-way	5 (83.3)
Two-way	1 (16.7)
Multi-way	1 (16.7)
Probabilistic	2 (33.3)
Multiple analysis	2 (33.3)
Addressed cost and outcome implications for scale
Yes	4 (66.7)
No	2 (33.3)

Five studies (83.3%) discounted the costs used in the analysis, but only one study (16.7%) reported a discount rate of 3%. The cost measure was consistent across studies, with the predominant outcome measure as QALYs in five studies (83.3%). Two studies (33.3%) reported the cost-effectiveness results in ICER, whilst the remaining used other cost-effectiveness measures. All studies performed at least one sensitivity analysis, 5 (83.3%). No study reported a cost-effectiveness threshold for concluding the app’s cost-effectiveness, and only four studies (66.7%) addressed the implications of the costs and outcomes for scaling the app intervention beyond the trial.

All studies reported significant methodological challenges in conducting CEA of the BCC apps. Limitations varied from lack of data to difficulties in valuing confounding effects. A summary of the limitations outlined in the CEA studies are categorized in Table 3.

Table 3.

Limitations in the CEA studies.

Study limitation	Number of studies affected (%)
Omitted healthcare sector, societal or both perspectives.	5 (83.3)
Difficulty with estimating results in ICER	2 (33.3)
No use of cost-effectiveness threshold(s)	6 (100%)
Lack of comparative cost-effectiveness data; false equivalency assumptions	3 (50.0)
Small sample size and/or losses to follow-up	4 (66.7)
Lack of (or limited access to) data	2 (33.3)
Difficulty with costing (e.g. micro versus macro)	4 (66.7)
Difficulty in estimating QALYs (or outcome effects or confounding effects)	5 (83.3)
Data extrapolation/modeling difficulty (assuming cost and outcome consistency over time)	1 (16.7)

Discussion

Our findings show a paucity of CEA studies of BCC apps and various challenges for conducting CEA studies involving BCC apps. This results suggest that the overall economic benefits of mHealth apps is often assumed and have thus far not been substantiated with sufficient empirical evidence. We hypothesized linkages between the challenges for conducting CEA that ultimately leads to poor-quality evidence and low comparability across CEA studies involving BCC apps (Figure 2). Barriers include data quality and availability. Inadequate cost and outcome measures, and insufficient data sources also represent important quality considerations which could ultimately affect ICER estimations. Overcoming these barriers can increase the transparency, consistency and transferability of CEA findings. ²⁷

Figure 2.

Hypothesized linkages between limitations for conducting CEA of BCC Apps.

Challenges for conducting CEA of BCC apps

Program effectiveness

Most studies used Randomized Controlled Trials (RCT) to estimate program effectiveness while others relied on effectiveness evidence from the literature or no evidence at all. Randomization is a major defense against unequal distribution of confounders. Thus, RCT remains the gold-standard for assessing effectiveness and efficacy in many mHealth interventions.³⁷ Restricting effectiveness assessments of BCC apps to RCTs is, however, problematic because it limits the scope of the evidence base by overlooking additional evidence from non-RCT methods. Furthermore, RCTs have a time lag of approximately 5.5 years between initiation of subject recruitment and publication of results which increases the risk for technology obsolescence of app functions. ³⁷ While there is a need to advocate for the recognition of non-RCT methods, rigorous approaches that balance fidelity with practicality are needed to ensure that quantifying app effects in economic evaluations is feasible. Three key methodological approaches for overcoming this barrier includes: 1) causal inference from observational data,^38,39 2) obtaining multiple repeated measures on a few participants, rather than a few measures on many participants; and 3) model-based designs for adaptive interventions, which requires an understanding of within-subject differences and the effect of mediating variables on health outcomes. ³⁷ These approaches will likely reduce the cost and time for data collection, and accommodate the continuously evolving technology in BCC app designs.

Attention should be given to small sample size and high losses to follow-up which can increase the uncertainty in cost and effect estimations, and introduce errors into scale analysis. ⁷ Methods for determining an accurate sample size varied among the studies, with sample size ranging from 123 to 1000 participants. For example, Dahlberg et al. used sample size of 1000 participants based on QALY weights in patients with an asymptomatic gallstone disease and a surgical scar, and assumptions of detecting a difference of 0.03 in QALYs between the groups. ³⁶ Conversely, Sjöström et al. used a convenience sample of 123, who were community dwelling women with stress urinary incontinence recruited via a website. ³⁵ Inconsistencies in calculating sample size for BCC app evaluation is challenging because a small sample size could indicate lower cost-savings, which may be higher with a larger sample size. Additionally, a small sample could represent a lack of user volume, which could indicate that more utilization may increase cost. ⁷ This problem was exemplified by Luxton et al. who used a sample size of 1000 participants to conclude that there was no cost-savings to using the “Breath2Relax” app. However, the app became less expensive compared to in-office treatment at approximately 1600 users. ⁴⁰ An appropriate sample size enhances economies of scale analysis, which is important for determining the costs and outcomes associated with the intervention if scaled to a larger population post-trial.

Analytical perspective

One study (16.7%) considered both societal and healthcare sector perspectives, while the remaining studies each considered a single perspective. To conduct CEA from a societal perspective, the Second Panel recommended the inclusion of an Impact Inventory. ²⁷ Though most of the studies predated this recommendation, our assessment was based on whether all relevant costs and effects were included in the studies’ analyses. The review showed that all studies had deficits in their costs and outcome estimations. While a narrower perspective accounts for costs and outcomes attributable to specific decision makers, it also represents a missed opportunity for evaluating the full cost and effects of the intervention. It was however uncertain whether a societal perspective which considered costs and outcomes regardless of the payer and beneficiary was better justified for any of the BCC apps. For example, Cano Martin et al. evaluated the CardioManager app intervention which required significant user input but only considered a healthcare sector perspective which ignored time costs attributable to the user. ⁴¹ A healthcare sector perspective was advantageous in this case due to its relevance to budget holders, but the omission of costs and outcomes attributable to the user constituted incomplete data which was important for decisions regarding large-scale implementation.

It is important to note that BCC apps vary widely in interventions, app features, and the targeted stages of behavior change. ³ In many cases, BCC apps are used outside the formal healthcare sector with cost savings accrued to the broader society. ⁷ This assertion was corroborated by Armstrong et al. who found that the higher cost of in-person follow-up care was spread between the healthcare system and patient, but the patient reaped the majority of cost-savings from participating in a mobile app follow-up care. ⁵ Furthermore, health interventions can have multiple unintended effects in targeted and untargeted participants over a short period of time. ²⁷ Thus, identifying and quantifying health costs and outcomes in different sectors can be challenging from a societal perspective. In cases where the “Rule of Reason” is applied, still identifying and valuing certain app costs and effects before ultimately judging that their exclusion has no significant impact on results can be time-consuming. While the Second Panel acknowledges these challenges, a more feasible measurement strategy for quantifying the external effects of BCC apps is yet to be identified.

Quantifying outcomes (effects)

It is critical to accurately identify and quantify intervention outcomes in CEA studies. ²⁷ Outcome measures varied across the studies, and the clinical/behavioral endpoints were inconsistent among the BCC apps. Consistent with the Second Panel’s recommendation, the predominant outcome measure used in the studies (83.3%) was QALYs. The QALY end-point combines quality of life (QoL) and survival outcome data, which provides comparability between studies. ²⁶ However, the QoL measure is a multi-dimensional health function usually obtained through preference weights associated with specific health states. Additionally, QALYs sum the total years lived, weighting each year by a quality of life weight between 0(dead) and 1(perfect health). ⁴² Obtaining preference weights for specific health states through a BCC app is complex, due to the need to collect detailed preference data while the app is in use. For example, Dahlberg et al. in evaluating the RAPP app did not find any difference in QALYs gained between the groups because the SF-6D measure used in estimating the QALYs functions were not affected by user follow-up routines. ³⁶ Overcoming this limitation is challenging because the endpoint for most BCC apps are intermediate outcomes which may not always be clinically quantifiable. Furthermore, estimating QALYs from intermediate outcomes requires assumptions which are not always backed with empirical data.

An additional impediment to estimating intermediate outcomes in BCC apps is the fact that they do not always lead to clinical or behavioral endpoints. Intermediate outcomes are common among health interventions aimed at health promotion and disease prevention because studies to estimate changes in long-term outcomes are costly and time-consuming to conduct. ⁴² As noted by the Second Panel, intermediate outcomes allows for studies with smaller sample sizes and shorter time frames because they are more likely to occur much sooner than the final desired clinical outcome(s). ²⁷ Furthermore, insufficient data remains a significant barrier to estimating the desired clinical endpoints from intermediate outcomes in BCC apps. Three proposed strategies for overcoming this barrier are to: 1) make the case that intermediate outcomes of BCC apps have value and clinical relevance in their own right, 2) prove that the link between an intermediate and final outcome (clinically-relevant endpoint) have been adequately established by previous research, or 3) address the uncertainties surrounding the link between intermediate and final outcomes in the analysis. ⁴² Despite these proposed strategies, questions about the usefulness and comparability of intermediate outcomes in CEA studies linger.

Health interventions could have confounding effects that need to be separated and quantified in order to estimate the true effects of the intervention. However, BCC apps can be confounded by external effects that are not always separable or quantifiable. For example, Reijen et al. noted that many athletes using the “Strengthen your Ankle” app may have already performed some sort of neuromuscular training (NMT) prior to participating in the trial, which may have reduced their initial risk for developing a recurrent ankle sprain. ³³ Quantifying the effects of the previous NMT to estimate the true effect of the app was impossible, although a previous NMT had significant impacts on the outcome of using the app. Methodological guidelines for accurately accounting for BCC app confounders is warranted to increase the transparency in estimating BCC app effects.

Quantifying costs

The Second Panel identified two types of costs associated with health interventions: 1) costs for production, delivery and consumption of the intervention, and 2) costs used or saved as a consequence of the intervention. Time is an important consideration in valuing short and long-term costs. With BCC apps, it is not feasible to wait for lifetime data to validate cost-effectiveness. Hence, the need for modelling techniques that can simulate costs and outcome effects over long periods beyond the trial. ⁷ To overcome the uncertainties of modeling, some analysts make costs and outcome consistency assumptions which may be flawed (Figure 2). This example was shown by Sjöström et al. who assumed that costs and utility weights measured at 3-month follow-up would remain constant over the year based on previous studies of internet-based pelvic floor muscle training where improvements after 3 months were maintained after 1 and 2 years. ³⁵ While this assumption may be reasonable, it does not account for the difference in user engagement or dependence on human resources. Consequently, sensitivity analyses are necessary to evaluate cost and effect uncertainties, the omission of which constitutes a bias. Consistent with the Second Panel’s recommendation, all the studies performed at least one sensitivity analysis to address uncertainties.

BCC apps can also have effects outside the health sector from changes in individual behaviors, and the associated costs or savings can accrue to the broader society. ²⁷ Identifying all costs attributable to the BCC app is challenging, and inconsistencies arise when decisions on what costs to include is at the analyst’s discretion based on the “Rule of Reason”. The major limitations in costing cited by Armstrong et al. were micro versus macro costing, and inadequate justifications for including present and future costs resulting from the app. ⁵ Micro-costing is a cost estimation method that includes the precise estimation of every input in the intervention, whereas macro-costing only considers broader-level costing whose omission will influence the overall cost. An appropriate costing method is important for deciding which cost data to include in order to increase transparency and transferability of CEA findings. Cost data used in the studies included estimates from accounting departments, patient cost databases, insurance billing codes and business plans. These data sources were inconsistent across studies, which could lead to incorrect assumptions in estimating the costs and effects of different BCC apps.

Lack of comparative data in CEA studies can result in low quality of cost measures (Figure 2). ⁴³ International health technology assessment agencies for economic evaluations recommend that health technology interventions be compared with their “usual care” alternatives to increase transparency in assessments. ⁴⁴ However, poor comparative data exacerbates problems with comparing BCC apps with their usual care comparators. This is because usual care treatments vary substantially in behavioral health interventions due to the lack of a ‘gold-standard’ treatment for specific health behaviors. ³⁵ Furthermore, cost-effectiveness data on “usual care” are scarce due to limited number of studies, and equivalency assumptions between interventions evaluated at different times should likewise be used with caution. This challenge is exemplified by Armstrong et al, who assumed equivalency in the effectiveness of mobile app and in-person follow-up care based on observational studies of telephone follow-up care from similar ambulatory surgical patients. ⁵ Since no RCT had demonstrated equal effectiveness between mobile app and in-person follow-up care, the BCC app could only be considered cost-effective in this context given the stated assumptions. The difficulty with generalizing evidence from evaluations in specific contexts, highlights the importance with balancing local applicability of BCC apps with the growing globalization of mHealth apps. It is imperative that assessments for scale recognize that a BCC app may be cost-effective in one context but highly expensive or ineffective in another context where access and resources are limited.

Estimating incremental cost-effectiveness ratio (ICER) in BCC apps

An appropriate comparison between two health interventions should be in terms of ICER. ⁴² Consistent with this recommendation, disaggregated cost and outcome measures are to be summarized into a single quantitative measure like ICER. ²⁷ A review of the studies showed no uniformity in estimating single quantitative measures for cost-effectiveness. For example, Van Reijen et al. expressed the cost-effectiveness of the Strengthen your Ankle app in “cost per injury incidence density”, ³³ whilst Amstrong et al. expressed cost-effectiveness of the BYOD app in “incremental net benefit”. ⁵ Likewise, Ryan et al. estimated the cost-effectiveness of the t+ Asthma app in “cost per mini-asthma quality of life”. ²¹ While estimating cost per QALY for BCC apps is arduous for analysts due to intermediate outcomes, the lack of a single quantitative measures for BCC apps has far reaching implications for decisions regarding adoption and scale. It also raises questions about the appropriate use of economic evaluation data in healthcare such as comparability with other studies, BCC app pricing, consumer willingness to pay, and potential health insurance reimbursements.

Another concern with estimating ICER lies with the appropriateness in using an ICER threshold, in which a cut-off point is considered for the cost per QALY ratio. The examples above illustrate the dilemma of cost-QALY thresholds when many BCC apps produce intermediate outcomes. The lower the ratio of a cost per QALY, the more cost-effective the health intervention is. Generally, the ICER has no theoretical or empirical basis, but values ranging from $50,000 to $100,000 are a considerable threshold for cost-effectiveness in the US. ⁴⁵ However, it is unclear whether the use of ICER thresholds in estimating the cost-effectiveness of BCC apps encapsulates all the relevant criteria a decision maker may require for adoption and implementation. While the ICER allows for easy comparison across studies, its applicability for resource allocation is not always straightforward. For example, BCC apps have the potential to reduce health inequities by increasing healthcare access to hard-to-reach populations. However, the use of costs and QALYs in estimating their cost-effectiveness does not automatically consider equity in healthcare access, which may be an important decision for adoption. Methods for incorporating important behavioral variables in estimating the cost-effectiveness of BCC apps are yet to be addressed.

Limitations

This study provides an important overview of the cost-effectiveness landscape of BCC apps, and highlights methodological challenges for conduction CEA. The methodological challenges highlighted may not be generalizable to all types of mHealth apps, although it can serve as a guide. We searched through four major databases to retrieve the final selected studies, and therefore it is possible that a few key studies may have been missed. Articles not published in English were also omitted.

Conclusion

Institutionalizing BCC apps into routine behavioral health intervention could likely increase their population health effects. However, evidence backing the economic value of BCC apps is limited. In this paper, we sought to highlight the economic evaluation characteristics of BCC apps and report limitations for conducting CEA using the Second Panel’s key guidelines. The review showed that economic evaluation studies on BCC apps attempt to quantify the costs and behavioral outcomes associated with using apps, but the majority of studies were fraught with study design challenges stemming from limited cost data and intermediate effects which sometimes have no clinically-relevant endpoints. Studies that use CEA methods also face challenges in adhering to recommended guidelines provided by the Second Panel. While economic evaluations are necessary to generate the evidence needed for scaling BCC apps, it is believed that more attention should first be given to the challenges in adhering to rigorous methodological guidelines. Furthermore, practical approaches that make conducting economic evaluations of BCC apps more feasible should be identified and prioritized. For example, economic evaluation methods should be linked to a set of appropriate measurement strategies and relevant data sources that factors in the unique attributes of BCC apps including intermediate outcomes and limited cost data. Only then can analysts pursue more complex economic assessments that can increase the transparency, quality and usefulness of the evidence. Until economic evaluations of BCC apps are conducted in accordance with methods that balance data completeness with practical concerns, decisions made on this body of evidence will be regarded with considerable caution.