Abstract
Background
Major depressive disorder is a chronic condition; its prevalence is expected to grow with the aging trend of high-income countries. Internet-based cognitive-behavioral therapy has proven efficacy in treating major depressive disorder.
Objective
The objective of this study was to assess the cost-effectiveness of implementing a community internet-based cognitive behavioral therapy intervention (Super@, the Spanish program for the MasterMind project) for treating major depressive disorder.
Methods
The cost-effectiveness of the Super@ program was assessed with the Monitoring and Assessment Framework for the European Innovation Partnership on Active and Healthy Ageing tool, using a 3-state Markov model. Data from the cost and effectiveness of the intervention were prospectively collected from the implementation of the program by a health care provider in Badalona, Spain; the corresponding data for usual care were gathered from the literature. The health states, transition probabilities, and utilities were computed using Patient Health Questionnaire–9 scores.
Results
The analysis was performed using data from 229 participants using the Super@ program. Results showed that the intervention was more costly than usual care; the discounted (3%) and nondiscounted incremental cost-effectiveness ratios were €29,367 and €26,484 per quality-adjusted life-year, respectively (approximately US $35,299 and $31,833, respectively). The intervention was cost-effective based on the €30,000 willingness-to-pay threshold typically applied in Spain (equivalent to approximately $36,060). According to the deterministic sensitivity analyses, the potential reduction of costs associated with intervention scale-up would reduce the incremental cost-effectiveness ratio of the intervention, although it remained more costly than usual care. A discount in the incremental effects up to 5% exceeded the willingness-to-pay threshold of €30,000.
Conclusions
The Super@ program, an internet-based cognitive behavioral therapy intervention for treating major depressive disorder, cost more than treatment as usual. Nevertheless, its implementation in Spain would be cost-effective from health care and societal perspectives, given the willingness-to-pay threshold of €30,000 compared with treatment as usual.
Keywords: digital health, telemedicine, eHealth, e–mental health, internet-based cognitive behavioral therapy, depression, iCBT, implementation, cost-effectiveness, cognitive behavioral therapy, CBT, cost
Introduction
Population aging is a global trend and is expected to be one of the most significant societal challenges worldwide in upcoming years [1]. The profound impact that this aging trend is likely to cause on our societies and economies has prompted significant efforts in turning the challenges of this scenario into opportunities for rethinking the way we design and organize our society, including the delivery of health and social care services [2-5].
Depression is a significant contributor to morbidity during entire lifespans and has been among the 3 leading nonfatal causes of disability globally for nearly three decades [6]. Although often underdiagnosed, depression is the most prevalent mental health condition among adult population and across cultural settings resulting in an aggregate point prevalence of 12.9%, 1-year prevalence of 7.2%, and lifetime prevalence of 10.8% (years 1994-2014) [7-9].
The burden of depression is specifically high among the elderly, irrespective of the presence of cognitive impairment, particularly in long-term care settings [8,10,11]. Various factors may increase the risk of depression among older people, including physiological factors (eg, cardiovascular disease, diabetes, or immunological changes) and psychosocial factors (eg, low economic status, social isolation, or relocation) [12-14]. Once established, depression in older people increases the risk of suicide and may trigger dementia [10].
While the efficacy of psychotherapy in the treatment of depression has been proven [15], the availability of evidence-based interventions constitutes a persistent challenge given the lack and unequal distribution of qualified practitioners, delayed provision of treatment, and inadequacy of treatment [16,17]. Given the limitations and health care costs associated with treating depression (eg, US $7638, according to a study conducted in Singapore [18]), there is growing interest in alternative therapies to routine care. Among them, a plethora of internet-based cognitive behavioral therapies for depression treatment have been introduced, many showing efficacy in treating major depressive disorder [19-21]. Although costs associated with the implementation of these therapies have been assessed, most studies are based on descriptive approaches, and formal cost-effectiveness analysis of internet-based cognitive behavioral therapy interventions are scarce [22].
While randomized controlled trials and accompanying cost-effectiveness analysis can be considered the gold standard in exploring the cost-effectiveness of mental health interventions, the idealized and controlled nature of these trials limits the generalizability of findings to routine care populations [23-25]. Establishing the cost-effectiveness of an intervention and its implementation under routine care conditions is an important part of the evaluation before wide-scale adoption. So far, establishing the cost-effectiveness of implementation projects in routine care provides a methodological challenge.
MasterMind was a European cofunded project aimed at scaling-up the implementation of evidence-based internet interventions (eg, internet-based cognitive behavioral therapy) for the treatment of adults experiencing depressive symptoms across Europe [26]. In this study, we assessed the cost-effectiveness of the Super@ intervention as part of its implementation within the MasterMind project in a pilot site in Spain. The current analysis was performed using the Monitoring and Assessment Framework for the European Innovation Partnership on Active and Healthy Ageing (MAFEIP) tool, developed for monitoring the financial sustainability of initiatives for promoting a healthy lifespan of European citizens [27,28]. Provided as a free-to-access tool for economic evaluations, MAFEIP has gained relevance over the years, and its usage is expanding, particularly within the European project landscape.
Methods
Overview of Study Design
As part of the MasterMind project for implementing an internet-based cognitive behavioral therapy for treating depression, we designed a pragmatic within-group trial to assess the cost-effectiveness of the intervention [29,30]. The evaluation framework applies the Model for Assessment of Telemedicine applications [31], which helped to define the data collection tools and instruments according to 3 levels of stakeholders involved within the implementation process: (1) patients, (2) professionals, and (3) organizations.
This analysis corresponds to the experience of the MasterMind project in the BSA (Badalona Serveis Assistencials) consortium, implemented under a program named Super@ tu depresión (“Get over your depression”). The BSA consortium provides primary and specialized care to a catchment population of 433,175 inhabitants in the most densely populated suburban area of Barcelona and has a long tradition in integrated care and the adoption of digital health solutions [32-38]. The implementation and data collection process for the Badalona pilot site was carried out between March 2015 and June 2017. The outcomes and costs of the intervention were compared with those of usual care in previously published data from the same area [39].
The local study protocol was approved by the Ethics Committee of the Hospital Germans Trias i Pujol (reference PI-15-069), and all participants provided informed consent before entering the study.
Participants
Study candidates included health care recipients and were screened for eligibility after general practitioner referral in the primary care setting. All consecutive patients who visited their general practitioners during the study period and met the selection criteria were offered the opportunity to participate in the Super@ program. Patients included in the study were adults (ie, 18 years or older) diagnosed with mild, moderate, or severe major depressive disorder based on the Patient Health Questionnaire 9 (PHQ-9; with score cutoffs of 10, 15, and 20 for mild, moderate, and severe major depressive disorder, respectively), living in Badalona and who, according to their general practitioner, had a certain level of technological literacy and internet connection. The main exclusion criteria were having comorbidities that may interfere with the treatment, having a nonpsychiatric disease that could explain depressive symptoms, receiving structured face-to-face psychological therapy at the time of inclusion, and reporting a high suicidal risk or ideation (item 9 of the PHQ-9). After checking all selection criteria and obtaining written informed consent, the general practitioner referred participants to the internet-based cognitive behavioral therapy service, provided a comprehensive explanation about the intervention, and enrolled participants in the platform, which automatically provided a username and a password to the participant.
Intervention
The Super@ program (Multimedia Appendix 1) consisted of 9 modules (8 regular and 1 extra) composed of videos, text content, and questionnaires to monitor the progression of symptoms and adherence to the intervention. Therapists provided guidance and project management within the BSA team to ensure patient follow-up and activation of the appropriate resources upon an increase of depressive symptoms. A project management team facilitated the project and its implementation process. Table S1 (Multimedia Appendix 1) summarizes the main activities performed in the project and the different professional profiles and teams involved in each. Intervention completion (ie, minimal adequate dose) was defined as engaging in a minimum of 3 modules of internet-based cognitive behavioral therapy.
Cost-Effectiveness Assessment
Model Structure, Transition Probabilities, and Utility Estimates
The cost-effectiveness of the Super@ program was assessed using the MAFEIP tool, which computes costs and utilities using a Markov model of health states and corresponding transition probabilities [40]. Based on previous economic evaluations of treatments for major depressive disorder, we defined a 3-state Markov model, with remission (PHQ-9 score <10), depression (PHQ-9 score ≥10), and death [41] (Figure 1). Transitions between the 3 states of the Markov model included recovery (ie, the probability of going from depression to remission) and relapse (ie, the probability of going from remission to depression); death was used as an absorbing state. The transition probabilities for the intervention group were calculated based on the changes between the health states at baseline and after the intervention. Given the lack of a control group, the corresponding probabilities for treatment as usual were obtained from a recent meta-analysis [41] assessing the usual care effects on major depressive disorder, which included 38 studies with pooled a remission rate (adjusted for publication bias) of 33% (95% CI 26%-40%). As suggested elsewhere [42], the risk of all-cause mortality was derived from life tables―in this case, the Human Mortality Database [43], which is stratified by gender and provides mortality rates at concrete years of age―and adjusted for depression [44].
Figure 1.
Diagram of the 3-state Markov model of health states and transition probabilities designed for the Super@ intervention.
In accordance with standard guidelines for estimating quality-adjusted life-years in economic evaluations, the MAFEIP tool recommends computing utilities based on measures of health-related quality of life, preferably the EuroQoL 5-dimension (EQ-5D) questionnaire [45]. However, no estimates of health-related quality of life were collected during the assessment of the Super@ program. Alternatively, based on the relationship between utility scores derived from (including the EQ-5D score) and depression health states reported by Kolovos et al [46], we estimated the remission utility from the results of the PHQ-9 measure: the values proposed for 4 clinical categories of major depressive disorder severity were adapted to the 3-state model by estimating the weighted average of utilities of patients in the remission state (ie, PHQ-9 score <10) and those in the depression state (PHQ-9 score ≥10) (Multimedia Appendix 1).
Cost Estimate
The MAFEIP tool considers 3 types of costs: one-off costs, which represent the total cost incurred only once at the implementation point (ie, implementation support, training provision of professional staff, and cost of devices), health care costs (ie, health care resources consumption such as costs associated with the time spent by health care professionals on service provision, hospitalizations, pharmacy, etc), and societal costs (ie, related to the time spent by either patients or informal caregivers such as the time spent using the technology or traveling to the hospital).
One-off costs were the main costs of implementing the Super@ services and included the support given to therapists to implement the intervention in their daily routine, the training of professional users, and the costs of development and adaptation of Super@ to the existing information and communication technology platform. Based on the annual gross salary of technical staff in Spain and the number of hours devoted to the project (ie, part-time 50%), we estimated €158 per patient (approximately US $190; an exchange rate of approximately €1 to US $1.20 is applicable at the time of publication). The costs of development and adaptation of Super@ to the existing information and communication technology platform were €237 per patient (US $285). Recurring costs, which included direct costs of each internet-based cognitive behavioral therapy session, amounted to €2439 (US $2927) per patient. For the control group, the typical situation is setting the one-off and recurrent costs at 0, because in most cases, the intervention would mean an additional investment.
Health care and societal costs were not collected in the MasterMind project. These costs were gathered from a previous study [39] that described the costs associated with major depressive disorder in the same area. Based on this study [39], we established the health care costs for patients in remission and depression as €451 and €826, respectively (US $542 and US $993). Correspondingly, the costs due to loss of labor productivity were €991 and €1842 for patients in remission and depression, respectively (US $1191 and US $2214). Health care and societal costs were assumed to be the same for the intervention and control groups.
In accordance with recommendations from local health technology assessment authorities in Spain, we applied a discount factor of 3% for both health care outcomes and costs [47]. The willingness-to-pay threshold was established at €30,000 per quality-adjusted life-year, the threshold most frequently used in Spain (equivalent to approximately $36,060).
The cycle length of the Markov model was set at 1 year (ie, the maximum allowed in the MAFEIP tool). Given the chronic nature of major depressive disorder [48], we established the number of cycles necessary to cover the time lapse between the average age of study participants (ie, 46 years) and a theoretical lifespan time of 100 years.
Analysis
The cost-effectiveness analysis was performed on the intention-to-treat sample, which included all participants who started at least 1 module of the treatment. The clinical and demographic characteristics of study participants were described with R software (version 3.5.3; The R Project) using the frequency percentage and the mean and standard deviation for categorical and quantitative variables, respectively. Variables of time were described as the median and interquartile range. The cost-effectiveness analysis was conducted using the MAFEIP tool including health states, transition probabilities, utility scores, and costs. All participants started on the state depression in the 3-state Markov model.
In addition to the base-case analysis, we conducted deterministic sensitivity analyses for 2 scenarios: reduction in session cost (up to 25%) associated with a lower professional-to-patient ratio expected for a scaling up of the intervention, and 0% to 5% discount in utilities, as recommended by local guidelines for economic evaluations [49]. Sensitivity analyses were nondiscounted.
Transition probabilities were computed using R software, whereas costs and utilities were calculated using a spreadsheet (Excel, version 2013; Microsoft Inc).
Results
Study Population and Intervention Conduct
Of the 253 patients recruited for the study, 229 participants (90.5%) started at least one module of the treatment (intention-to-treat sample), of whom 1 participant (0.4%) did not provide data on posttreatment status, and 81 participants (35.4%) did not complete treatment; therefore, 147 participants completed the treatment (Multimedia Appendix 1). All participants had been recruited during a clinical interview after referral by their general practitioner, and all completed the PHQ-9 questionnaire. Table 1 summarizes demographic and clinical characteristics of the intention-to-treat sample at baseline. Participants in the intention-to-treat sample remained under the Super@ program a median of 96 days (IQR 70-321); 147 participants (64.2%) were considered to have completed the study. At the end of the intervention, 98 participants (66.7%) had achieved the remission state. No adverse events related to the intervention or the major depressive disorder were reported.
Table 1.
Clinical and demographic baseline characteristics of the participants who started the treatment.
| Characteristic | Intention-to-treat sample (n=229) | |
| Age (years), mean (SD) | 46.40 (12.51) | |
| Gender, n (%) |
|
|
|
|
Male | 73 (31.9) |
|
|
Female | 156 (68.1) |
| Education, n (%) |
|
|
|
|
Primary | 42 (18.3) |
|
|
Secondary | 100 (43.7) |
|
|
Higher | 78 (34.1) |
|
|
Other | 8 (3.5) |
|
|
Not answered | 1 (0.4) |
| Employment, n (%) |
|
|
|
|
Yes | 169 (73.8) |
|
|
No | 58 (25.3) |
|
|
Unknown | 1 (0.4) |
|
|
Not answered | 1 (0.4) |
| Depressive episodes, n (%) |
|
|
|
|
Less than 4 weeks | 10 (4.4) |
|
|
Between 4 and 8 weeks | 40 (17.5) |
|
|
Between 8 and 12 weeks | 65 (28.4) |
|
|
Between 3 and 6 months | 51 (22.3) |
|
|
Between 6 months and 1 year | 36 (15.7) |
|
|
Between 1 year and 3 years | 23 (10.0) |
|
|
Between 3 and 5 years | 2 (0.9) |
|
|
Between 5 and 10 years | 2 (0.9) |
| Antidepressant medication, n (%) |
|
|
|
|
Yes, for less than 1 month | 7 (3.1) |
|
|
Yes, for less than 2 months | 44 (19.2) |
|
|
Yes, for more than 2 months | 74 (32.3) |
|
|
No | 104 (45.4) |
| Satisfaction with lifea, n (%) |
|
|
|
|
Preintervention | 3.50 (1.16) |
|
|
Postintervention | 4.03 (1.28) |
| Satisfaction with mental healtha, n (%) |
|
|
|
|
Preintervention | 3.23 (1.03) |
|
|
Postintervention | 3.98 (1.32) |
aAssessed using a single-item question (How satisfied are you with your life as a whole today? or How satisfied are you with your mental health?) and rated on a 6-point scale (1=couldn’t be worse, 2=displeased, 3=mostly dissatisfied, 4=mixed, 5=mostly satisfied, 6=pleased, 7=couldn’t be better).
Study Parameters and Base Case Analysis
Table 2 summarizes the inputs of the cost-effectiveness analysis, including transition probabilities, costs, and utilities.
Table 2.
Inputs of the MAFEIP tool for computing the cost-effectiveness analysis.
| Input | Control group | Intervention group (n=229) | |
| Transition probabilities (%) |
|
|
|
|
|
Remission | 14 | 0 |
|
|
Depression | 29 | 48.53 |
| Costs (€a per patient and year) |
|
|
|
| One‐off cost per patient | N/Ab | 395.26 | |
| Recurring cost per patient per year | N/A | 2439 | |
| Health care cost |
|
|
|
|
|
Remission | 451 | 451 |
|
|
Depression | 826 | 826 |
| Societal cost |
|
|
|
|
|
Remission | 991 | 991 |
|
|
Depression | 1842 | 1842 |
| Utilities |
|
|
|
|
|
Remission | 0.62 | 0.665 |
|
|
Depression | 0.532 | 0.529 |
| Relative risk of mortality |
|
|
|
|
|
Remission | 1 | 1 |
|
|
Depression | 1.68 | 1.68 |
aAn exchange rate of approximately €1 to US $1.20 is applicable at the time of publication.
bN/A: not applicable.
The Super@ program cost more than usual care from both health care and societal perspectives (Table 3).
Table 3.
Incremental costs, effects, and cost-effectiveness ratio from health care and societal perspectives.
| Perspective | Incremental cost (€a) | Incremental effects (QALYb) | Incremental cost-effectiveness ratio (€/QALY) | |
| Health care perspective |
|
|
|
|
|
|
Discounted (3% for both costs and effects) | 50,924.53 | 1.734 | 29,366.92 |
|
|
Nondiscounted | 87,807.06 | 3.315 | 26,484.27 |
| Societal perspective |
|
|
|
|
|
|
Discounted (3% for both costs and effects) | 48,178.53 | 1.734 | 27,783.38 |
|
|
Nondiscounted | 83,181.81 | 3.315 | 25,089.21 |
aAn exchange rate of approximately €1 to US $1.20 is applicable at the time of publication.
bQALY: quality-adjusted life-year.
The nondiscounted incremental cost-effectiveness ratios were below the willingness-to-pay threshold of €30,000 (Figure 2): €26,484 and €25,089 for health care and societal perspectives, respectively (US $31,833 and $30,162). The discounted incremental costs and effects were higher, although the incremental cost-effectiveness ratios remained below the willingness-to-pay threshold of €30,000.
Figure 2.
Cost-effectiveness plane of the Super@ intervention Healthcare perspective discounted (3% for both costs and health effects) (A) and non-discounted (B) analyses. Societal perspective discounted (3% for both costs and health effects) (C) and non-discounted (D) analyses. The solid line shows the 30,000 €/QALY willingness-to-pay threshold (equivalent to approximately US $36,060; an exchange rate of approximately €1 to US $1.20 is applicable at the time of publication). QALY: quality-adjusted life-year. WTP: willingness-to-pay.
In addition, we conducted a deterministic sensitivity analysis assuming that a greater number of participants to the program would results in a reduction of cost per session. A 25% reduction in the cost per session would reduce the incremental cost-effectiveness ratio from €26,484 to €19,623 (US $31,833 to $23,591) in the health care perspective analysis and from €25,089 to €18,228 (US $30,162 to $21,914) in the societal perspective analysis (Figure 3A and 3B). From the health care perspective (Figure 3C), the incremental cost-effectiveness ratio at the 5% discount in utility (worst-case scenario of the sensitivity analysis) was €71,041 (US $85,405). The corresponding intersection and lowest incremental cost-effectiveness ratio values for the societal perspective were 2.773 quality-adjusted life-years and €30,000 (Figure 3D).
Figure 3.
Cost-effectiveness planes of sensitivity analyses. A reduction of up to 25% in cost per session (A and B for healthcare and societal perspectives, respectively), and 0% to 5% discount in effects (C and D for healthcare and societal perspectives, respectively). The dotted black line shows the 30,000 €/QALY willingness-to-pay threshold (equivalent to approximately US $36,060; an exchange rate of approximately €1 to US $1.20 is applicable at the time of publication). The solid green line shows the range of the incremental cost-effectiveness ratio, with the red and green squares indicating the range extremes for the worse (more costly or less effective) and best (less costly or more effective) scenario, respectively. QALY: quality-adjusted life-year. WTP: willingness-to-pay.
Discussion
In this analysis of the cost-effectiveness of an internet-based cognitive behavioral therapy intervention for mild or moderate major depressive disorder, we found that the intervention was more effective than treatment as usual, with incremental costs of €87,807 and €83,181 (nondiscounted from the health care and societal perspectives, respectively; US $105,561 and $99,999), according to costs reported for routine care of patients with mild-to-moderate major depressive disorder in our area (Badalona, Spain). Despite the higher cost of the internet-based cognitive behavioral therapy intervention, it remained below the willingness-to-pay threshold of €30,000 typically used in Spain for making decisions in health care policies. According to the sensitivity analyses, the internet-based cognitive behavioral therapy would remain more expensive and more effective than treatment as usual in the onset of the cost reduction expected when scaling up the intervention (with the consequent decrease of the professional-to-patient ratio), with an incremental cost-effectiveness ratio below the willingness-to-pay threshold. When considering a 5% reduction in utility (ie, as suggested by local guidelines for economic evaluations), the intervention remained more effective than treatment as usual, although with an incremental cost-effectiveness ratio above the willingness-to-pay threshold.
In the last decade, many studies [50-53] have investigated the costs associated with internet-based cognitive behavioral therapy interventions, including therapies for major depressive disorder; however, most are based on descriptive approaches, which preclude drawing conclusions that can be used for making decisions on their implementation. More recently, Paganini et al [22] reviewed economic evaluations of internet-based cognitive behavioral therapy interventions for major depressive disorder that fulfilled preselected quality criteria, including the presence of comparator groups such as treatment as usual, another intervention, or wait-list controls. The case-mix of these interventions and heterogeneity of analyses precludes direct comparisons regarding the cost-effectiveness of each intervention. Nevertheless, they found that guided interventions (such as the Super@ program) tended to be more cost-effective than self-help ones, despite the higher cost associated with professional honoraria [22]. The incremental cost-effectiveness ratio of our intervention for the base-case health care perspective (€26,484 per quality-adjusted life-year) was in the lower zone of the wide range of values reported for guided interventions (ie, €19,616 [54] to €157,900 [55]; approximately US $19,616 and $189,825, respectively) and below that of unguided interventions (ie, €40,412 [56] to €178,700 [57]; approximately US $48,583 and $214,831, respectively).
Additionally, such studies [51,52] can only report on cost-effectiveness measures in controlled settings. Our study focused on the assessment of cost-effectiveness under real-world conditions free from biases possibly being introduced within efficacy studies such as a stricter application of protocolized procedures, eligibility criteria, and randomization [23-25]. Nevertheless, this approach resulted in some disadvantages, and our results should be interpreted with caution due to several limitations.
The lack of a comparator group has been considered among the limitations of economic evaluations of internet-based cognitive behavioral therapy interventions for major depressive disorder [22]. The pragmatic approach of our study, which took advantage of the implementation of the Super@ program by the local health care provider to assess its cost-effectiveness, precluded collecting treatment-as-usual data in parallel with those collected for the internet-based cognitive behavioral therapy intervention; however, the MAFEIP tool allowed us to rely upon literature for gathering these data. Of note, the source of cost-estimate data of treatment as usual for major depressive disorder corresponded to the same area in which the Super@ program was deployed [39]. Hence, the costs attributed to treatment as usual are expected to be similar to those we would have observed in a control group.
The MAFEIP tool also allowed us to bypass the unavailability of EQ-5D scores of health-related quality of life, a widely accepted measurement for computing utilities in cost-effectiveness analyses [45,58]. Other measures, such as disease severity scores, have been proposed as a proxy for health-related quality of life [59]. Taking advantage of the analysis by Kolovos et al, who established a relationship between health-related quality of life and PHQ-9 score for major depressive disorder severity [46], we computed the utility of the remission state of our 3-state Markov model using the PHQ-9 scores at the cutoff for minor depressive symptoms in the 5-state scale defined by the American Psychiatric Association [60] and the National Institute for Clinical Excellence [61].
Readers should take into consideration some limitations of the study design. First, the pragmatic approach constrained the number of participants to the implementation capacity, rather than the adequate sample size to achieve precision in our estimates. Second, like many other information and communication technology-based solutions, the success of an internet-based cognitive behavioral therapy intervention requires minimal technological literacy, which in our intervention was measured in an unstructured way at each general practitioners discretion. Technological literacy and keenness for the use of digital gadgets are expected to influence not only adherence but also the benefit that the patient may obtain from the intervention; the unstructured assessment of digital literacy may have introduced heterogeneity in the intervention outcomes. Third, the transferability of the results to other settings should be considered carefully. There are many reasons why cost-effectiveness analysis of health technologies may differ across jurisdictions and researchers and implementers should always refer to national guidelines in order to shed some light on the applicability of the results emerging from other contexts [62].
Our results suggest that the Super@ program provided benefits to patients at a cost that would allow its implementation in Spain, where interventions below €30,000 per quality-adjusted life-year are accepted. Costs associated with the intervention are expected to decrease in a scaling-up scenario; however, the sensitivity analysis of utility indicates that small reductions in effects would place the intervention at a nonacceptable incremental cost-effectiveness ratio based on the €30,000 threshold. Future studies should explore the patient profile that can benefit most from the intervention so that general practitioners have more information to target the therapy adequately.
Acknowledgments
The authors would like to thank Gerard Carot-Sans for providing medical writing support during the preparation of the manuscript. The authors would also like to thank the MasterMind project consortium partners for their contribution along the project.
The MasterMind project was partially funded under the Information and Communication Technology Policy Support Programme as part of the Competitiveness and Innovation Framework Programme by the European Community (grant agreement 621000).
The continuous development of the MAFEIP tool is funded by the European Commission’s Horizon 2020 program under the projects WE4AHA (grant number 769705) and DigitalHealthEurope (grant number 826353).
The funders had no role in data collection or analysis, the decision to publish, or the preparation of the manuscript.
Abbreviations
- BSA
Badalona Serveis Assistencials
- EQ-5D
EuroQoL 5-dimension questionnaire
- MAFEIP
Monitoring and Assessment Framework for the European Innovation Partnership (on Active and Healthy Ageing).
- PHQ-9
Patient Health Questionnaire–9
Appendix
Supplementary file 1.
Footnotes
Authors' Contributions: JPJ was the principal investigator at the Badalona pilot site for the MasterMind project. JPJ and AE contributed to the study design and data collection. JPJ and AE conducted the statistical analyses. JPJ drafted the manuscript. AE and SK contributed to the different versions of the paper. FLV and FF supervised the study. All authors critically revised and approved the final version of the manuscript.
Conflicts of Interest: Author AE is employed by the Institute for health training online as research coordinator. All other authors declare no competing interests.
References
- 1.Khan HTA. Population ageing in a globalized world: Risks and dilemmas? J Eval Clin Pract. 2019 Oct;25(5):754–760. doi: 10.1111/jep.13071. [DOI] [PubMed] [Google Scholar]
- 2.Beard JR, Bloom DE. Towards a comprehensive public health response to population ageing. Lancet. 2015 Feb 14;385(9968):658–661. doi: 10.1016/S0140-6736(14)61461-6. http://europepmc.org/abstract/MED/25468151. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Bloom DE, Canning D, Fink G. Implications of population ageing for economic growth. Oxford Rev Econ Policy. 2011 May 13;26(4):583–612. doi: 10.1093/oxrep/grq038. [DOI] [Google Scholar]
- 4.Hantrais L. Socio-demographic change, policy impacts and outcomes in social Europe. J Eur Soc Policy. 2016 Dec 21;9(4):291–309. doi: 10.1177/a010186. [DOI] [Google Scholar]
- 5.Fougère M, Mérette M. Population ageing and economic growth in seven OECD countries. Econ Model. 1999 Aug;16(3):411–427. doi: 10.1016/s0264-9993(99)00008-5. [DOI] [Google Scholar]
- 6.GBD 2017 DiseaseInjury IncidencePrevalence Collaborators Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet. 2018 Nov 10;392(10159):1789–1858. doi: 10.1016/S0140-6736(18)32279-7. https://linkinghub.elsevier.com/retrieve/pii/S0140-6736(18)32279-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bruce ML, Sirey JA. Integrated care for depression in older primary care patients. Can J Psychiatry. 2018 Jul;63(7):439–446. doi: 10.1177/0706743718760292. http://europepmc.org/abstract/MED/29495883. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Polyakova M, Sonnabend N, Sander C, Mergl R, Schroeter M, Schroeder J, Schönknecht P. Prevalence of minor depression in elderly persons with and without mild cognitive impairment: a systematic review. J Affect Disord. 2014 Jan;152-154:28–38. doi: 10.1016/j.jad.2013.09.016. [DOI] [PubMed] [Google Scholar]
- 9.Lim GY, Tam WW, Lu Y, Ho CS, Zhang MW, Ho RC. Prevalence of depression in the community from 30 countries between 1994 and 2014. Sci Rep. 2018 Feb 12;8(1):2861. doi: 10.1038/s41598-018-21243-x. doi: 10.1038/s41598-018-21243-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Alexopoulos GS. Depression in the elderly. Lancet. 2005;365(9475):1961–70. doi: 10.1016/S0140-6736(05)66665-2. [DOI] [PubMed] [Google Scholar]
- 11.Husain SF, Yu R, Tang T, Tam WW, Tran B, Quek TT, Hwang S, Chang CW, Ho CS, Ho RC. Validating a functional near-infrared spectroscopy diagnostic paradigm for major depressive disorder. Sci Rep. 2020 Jun 16;10(1):9740. doi: 10.1038/s41598-020-66784-2. doi: 10.1038/s41598-020-66784-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Ho R, Chua A, Tran B, Choo C, Husain S, Vu G, McIntyre R, Ho C. Factors associated with the risk of developing coronary artery disease in medicated patients with major depressive disorder. Int J Environ Res Public Health. 2018 Sep 21;15(10):2073. doi: 10.3390/ijerph15102073. https://www.mdpi.com/resolver?pii=ijerph15102073. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Vu HTT, Nguyen TX, Nguyen HTT, Le TA, Nguyen TN, Nguyen AT, Nguyen TTH, Nguyen HL, Nguyen CT, Tran BX, Latkin CA, Pham T, Zhang MW, Ho RC. Depressive symptoms among elderly diabetic patients in Vietnam. Diabetes Metab Syndr Obes. 2018;11:659–665. doi: 10.2147/DMSO.S179071. doi: 10.2147/DMSO.S179071. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Ng A, Tam WW, Zhang MW, Ho CS, Husain SF, McIntyre RS, Ho RC. IL-1β, IL-6, TNF- α and CRP in elderly patients with depression or Alzheimer's disease: systematic review and meta-analysis. Sci Rep. 2018 Aug 13;8(1):12050. doi: 10.1038/s41598-018-30487-6. doi: 10.1038/s41598-018-30487-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Cuijpers P, Karyotaki E, de Wit Leonore, Ebert D. The effects of fifteen evidence-supported therapies for adult depression:a meta-analytic review. Psychother Res. 2020 Mar;30(3):279–293. doi: 10.1080/10503307.2019.1649732. [DOI] [PubMed] [Google Scholar]
- 16.Wittchen H, Jacobi F, Rehm J, Gustavsson A, Svensson M, Jönsson B, Olesen J, Allgulander C, Alonso J, Faravelli C, Fratiglioni L, Jennum P, Lieb R, Maercker A, van Os J, Preisig M, Salvador-Carulla L, Simon R, Steinhausen H. The size and burden of mental disorders and other disorders of the brain in Europe 2010. Eur Neuropsychopharmacol. 2011 Sep;21(9):655–79. doi: 10.1016/j.euroneuro.2011.07.018. [DOI] [PubMed] [Google Scholar]
- 17.Mack S, Jacobi F, Gerschler A, Strehle J, Höfler Michael, Busch Markus A, Maske Ulrike E, Hapke Ulfert, Seiffert Ingeburg, Gaebel Wolfgang, Zielasek Jürgen, Maier Wolfgang, Wittchen Hans-Ulrich. Self-reported utilization of mental health services in the adult German population--evidence for unmet needs? results of the DEGS1-mental health module. Int J Methods Psychiatr Res. 2014 Sep;23(3):289–303. doi: 10.1002/mpr.1438. http://europepmc.org/abstract/MED/24687693. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Ho R, Mak K, Chua A, Ho C, Mak A. The effect of severity of depressive disorder on economic burden in a university hospital in Singapore. Expert Rev Pharmacoecon Outcomes Res. 2013 Aug;13(4):549–59. doi: 10.1586/14737167.2013.815409. doi: 10.1586/14737167.2013.815409. [DOI] [PubMed] [Google Scholar]
- 19.Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. 2012 Jun;32(4):329–42. doi: 10.1016/j.cpr.2012.02.004. [DOI] [PubMed] [Google Scholar]
- 20.Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther. 2009;38(4):196–205. doi: 10.1080/16506070903318960. [DOI] [PubMed] [Google Scholar]
- 21.Zhang M, Ho R. Moodle: the cost effective solution for internet cognitive behavioral therapy (I-CBT) interventions. Technol Health Care. 2017;25(1):163–165. doi: 10.3233/THC-161261. [DOI] [PubMed] [Google Scholar]
- 22.Paganini S, Teigelkötter Wiebke, Buntrock C, Baumeister H. Economic evaluations of internet- and mobile-based interventions for the treatment and prevention of depression: a systematic review. J Affect Disord. 2018 Jan 01;225:733–755. doi: 10.1016/j.jad.2017.07.018. [DOI] [PubMed] [Google Scholar]
- 23.Wells K B. Treatment research at the crossroads: the scientific interface of clinical trials and effectiveness research. Am J Psychiatry. 1999 Jan;156(1):5–10. doi: 10.1176/ajp.156.1.5. [DOI] [PubMed] [Google Scholar]
- 24.Rothwell P. External validity of randomised controlled trials: "to whom do the results of this trial apply?". Lancet. 2005;365(9453):82–93. doi: 10.1016/S0140-6736(04)17670-8. [DOI] [PubMed] [Google Scholar]
- 25.Singal AG, Higgins PDR, Waljee AK. A primer on effectiveness and efficacy trials. Clin Transl Gastroenterol. 2014 Jan 02;5:e45. doi: 10.1038/ctg.2013.13. http://europepmc.org/abstract/MED/24384867. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Management of mental health disorders through advanced technology and services – telehealth for the mind. MasterMind Project. [2021-05-07]. https://mastermind-project.eu/
- 27.Monitoring and assessment framework for the European innovation partnership on active and healthy ageing. European Commission. [2021-05-07]. https://www.mafeip.eu/
- 28.Boehler CE, de Graaf G, Steuten L, Yang Y, Abadie F. Development of a web-based tool for the assessment of health and economic outcomes of the European Innovation Partnership on Active and Healthy Ageing (EIP on AHA) BMC Med Inform Decis Mak. 2015 Sep 11;15(S3):1–1. doi: 10.1186/1472-6947-15-s3-s4. http://www.biomedcentral.com/1472-6947/15/S3/S4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Campbell DT, Stanley JC. Experimental and Quasi-experimental Designs for Research. Chicago: Rand McNally & Company; 2015. [Google Scholar]
- 30.Vis C, Kleiboer A, Prior R, Bønes E, Cavallo M, Clark SA, Dozeman E, Ebert D, Etzelmueller A, Favaretto G, Zabala AF, Kolstrup N, Mancin S, Mathiassen K, Myrbakk VN, Mol M, Jimenez JP, Power K, van Schaik A, Wright C, Zanalda E, Pederson CD, Smit J, Riper H. Implementing and up-scaling evidence-based eMental health in Europe: he study protocol for the MasterMind project. Internet Interv. 2015 Nov;2(4):399–409. doi: 10.1016/j.invent.2015.10.002. [DOI] [Google Scholar]
- 31.Kidholm K, Ekeland A, Jensen L, Rasmussen J, Pedersen C, Bowes A, Flottorp Sa, Bech M. A model for assessment of telemedicine applications: MAST. Int J Technol Assess Health Care. 2012 Jan 23;28(1):44–51. doi: 10.1017/s0266462311000638. https://www.cambridge.org/core/article/model-for-assessment-of-telemedicine-applications-mast/4C1ACDC52AA683BBCEACCBBA182D5F24. [DOI] [PubMed] [Google Scholar]
- 32.Baltaxe E, Cano I, Herranz C, Barberan-Garcia A, Hernandez C, Alonso Albert, Arguis María José, Bescos Cristina, Burgos Felip, Cleries Montserrat, Contel Joan Carles, de Batlle Jordi, Islam Kamrul, Kaye Rachelle, Lahr Maarten, Martinez-Palli Graciela, Miralles Felip, Moharra Montserrat, Monterde David, Piera Jordi, Ríos José, Rodriguez Nuria, Ron Reut, Rutten-van Mölken Maureen, Salas Tomas, Santaeugenia Sebastià, Schonenberg Helen, Solans Oscar, Torres Gerard, Vargiu Eloisa, Vela Emili, Roca Josep. Evaluation of integrated care services in Catalonia: population-based and service-based real-life deployment protocols. BMC Health Serv Res. 2019 Jun 11;19(1):370. doi: 10.1186/s12913-019-4174-2. https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-019-4174-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Dueñas-Espín Ivan, Vela E, Pauws S, Bescos C, Cano I, Cleries Montserrat, Contel Joan Carles, de Manuel Keenoy Esteban, Garcia-Aymerich Judith, Gomez-Cabrero David, Kaye Rachelle, Lahr Maarten M H, Lluch-Ariet Magí, Moharra Montserrat, Monterde David, Mora Joana, Nalin Marco, Pavlickova Andrea, Piera Jordi, Ponce Sara, Santaeugenia Sebastià, Schonenberg Helen, Störk Stefan, Tegner Jesper, Velickovski Filip, Westerteicher Christoph, Roca Josep. Proposals for enhanced health risk assessment and stratification in an integrated care scenario. BMJ Open. 2016 Apr 15;6(4):e010301. doi: 10.1136/bmjopen-2015-010301. https://bmjopen.bmj.com/lookup/pmidlookup?view=long&pmid=27084274. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Rossi Mori A, Albano V, Piera Jimenez J. Badalona story: integrating the integration initiatives. Int J Integr Care. 2017 Oct 17;17(5):315. doi: 10.5334/ijic.3632. [DOI] [Google Scholar]
- 35.RossiMori A, Piera J, Albano V, Mercurio G. A systematic analysis of the multi-annual journey of Badalona towards integrated care. Int J Integr Care. 2019 Aug 08;19(4):344. doi: 10.5334/ijic.s3344. [DOI] [Google Scholar]
- 36.Moharra M, Vela E, Dueñas-Espín I, Pauws S, Bescos C, Cano I, Cleries M, Contel JC, De Manuel Keenoy E, Garcia-Aymerich J, Gómez-Cabrero D, Kaye R, Lahr MM, Lluch-Ariet M, Monterde D, Mora J, Nalin M, Pavlickova A, Piera J, Ponce S, Santaeugenia S, Schonenberg H, Störk S, Tegnér J, Velickovski F, Westerteicher C, Roca J. Health risk assessment and stratification in an integrated care scenario. Int J Integr Care. 2016 Dec 16;16(6):322. doi: 10.5334/ijic.2870. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Piera-Jiménez Jordi, Winters M, Broers E, Valero-Bover D, Habibovic M, Widdershoven JWMG, Folkvord F, Lupiáñez-Villanueva Francisco. Changing the health behavior of patients with cardiovascular disease through an electronic health intervention in three different countries: cost-effectiveness study in the do cardiac health: advanced new generation ecosystem (Do CHANGE) 2 randomized controlled trial. J Med Internet Res. 2020 Jul 28;22(7):e17351. doi: 10.2196/17351. https://www.jmir.org/2020/7/e17351/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Piera-Jiménez Jordi, Daugbjerg S, Stafylas P, Meyer I, Müller Sonja, Lewis L, da Col P, Folkvord F, Lupiáñez-Villanueva Francisco. BeyondSilos, a telehealth-enhanced integrated care model in the domiciliary setting for older patients: observational prospective cohort study for effectiveness and cost-effectiveness assessments. JMIR Med Inform. 2020 Oct 06;8(10):e20938. doi: 10.2196/20938. https://medinform.jmir.org/2020/10/e20938/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Sicras-Mainar A, Mauriño Jorge, Cordero L, Blanca-Tamayo M, Navarro-Artieda R. [Costs and associated factors with optimal and suboptimal responses to the treatment of major depressive disorder] Aten Primaria. 2012 Nov;44(11):667–75. doi: 10.1016/j.aprim.2012.04.007. https://linkinghub.elsevier.com/retrieve/pii/S0212-6567(12)00210-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Weinstein M, O'Brien Bernie, Hornberger J, Jackson J, Johannesson M, McCabe Chris, Luce Bryan R, ISPOR Task Force on Good Research Practices--Modeling Studies Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices--Modeling Studies. Value Health. 2003;6(1):9–17. doi: 10.1046/j.1524-4733.2003.00234.x. https://linkinghub.elsevier.com/retrieve/pii/234. [DOI] [PubMed] [Google Scholar]
- 41.Kolovos S, van Tulder MW, Cuijpers P, Prigent A, Chevreul K, Riper H, Bosmans JE. The effect of treatment as usual on major depressive disorder: a meta-analysis. J Affect Disord. 2017 Mar 01;210:72–81. doi: 10.1016/j.jad.2016.12.013. [DOI] [PubMed] [Google Scholar]
- 42.Sonnenberg F, Beck J. Markov models in medical decision making: a practical guide. Med Decis Making. 1993;13(4):322–38. doi: 10.1177/0272989X9301300409. [DOI] [PubMed] [Google Scholar]
- 43.Jdanov D, Jasilionis D, Shkolnikov V, Barbieri M. Human mortality database. In: Gu D, Dupre M, editors. Encyclopedia of Gerontology and Population Aging. Cham: Springer; 2019. [Google Scholar]
- 44.Cuijpers P, Vogelzangs N, Twisk J, Kleiboer A, Li J, Penninx BW. Comprehensive meta-analysis of excess mortality in depression in the general community versus patients with specific illnesses. Am J Psychiatry. 2014 Apr;171(4):453–62. doi: 10.1176/appi.ajp.2013.13030325. [DOI] [PubMed] [Google Scholar]
- 45.NICE Position statement on EQ-5D-5L valuation set. Pharmaco Econ Outcomes News. 2017 Aug 26;785(1):7–7. doi: 10.1007/s40274-017-4257-4. https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-technology-appraisal-guidance/eq5d5l_nice_position_statement.pdf. [DOI] [Google Scholar]
- 46.Kolovos S, Bosmans J, van Dongen Johanna M, van Esveld Birre, Magai D, van Straten Annemieke, van der Feltz-Cornelis Christina, van Steenbergen-Weijenburg Kirsten M, Huijbregts Klaas M, van Marwijk Harm, Riper Heleen, van Tulder Maurits W. Utility scores for different health states related to depression: individual participant data analysis. Qual Life Res. 2017 Jul;26(7):1649–1658. doi: 10.1007/s11136-017-1536-2. http://europepmc.org/abstract/MED/28260149. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.López-Bastida Julio, Oliva J, Antoñanzas Fernando, García-Altés Anna, Gisbert R, Mar J, Puig-Junoy J. Spanish recommendations on economic evaluation of health technologies. Eur J Health Econ. 2010 Oct 20;11(5):513–20. doi: 10.1007/s10198-010-0244-4. [DOI] [PubMed] [Google Scholar]
- 48.Kolovos S, Bosmans JE, Riper H, Chevreul K, Coupé VMH, van Tulder MW. Model-based economic evaluation of treatments for depression: a systematic literature review. Pharmacoecon Open. 2017 Feb 7;1(3):149–165. doi: 10.1007/s41669-017-0014-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Mora-Ripoll R, Gilabert-Perramon A, Oliva-Moreno J, Puig-Junoy J. Guidance for economic evaluation and budget impact analysis for pharmaceuticals in Catalonia (Spain) Value Health. 2014 Nov;17(7):A447. doi: 10.1016/j.jval.2014.08.1193. https://linkinghub.elsevier.com/retrieve/pii/S1098-3015(14)03123-4. [DOI] [PubMed] [Google Scholar]
- 50.Tate D, Finkelstein E, Khavjou O, Gustafson A. Cost effectiveness of internet interventions: review and recommendations. Ann Behav Med. 2009 Aug;38(1):40–5. doi: 10.1007/s12160-009-9131-6. http://europepmc.org/abstract/MED/19834778. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Hedman E, Ljótsson Brjánn, Lindefors N. Cognitive behavior therapy via the Internet: a systematic review of applications, clinical efficacy and cost-effectiveness. Expert Rev Pharmacoecon Outcomes Res. 2012 Dec;12(6):745–64. doi: 10.1586/erp.12.67. [DOI] [PubMed] [Google Scholar]
- 52.Donker T, Blankers M, Hedman E, Ljótsson B, Petrie K, Christensen H. Economic evaluations of internet interventions for mental health: a systematic review. Psychol Med. 2015 Aug 03;45(16):3357–3376. doi: 10.1017/s0033291715001427. [DOI] [PubMed] [Google Scholar]
- 53.Arnberg F, Linton S, Hultcrantz M, Heintz E, Jonsson U. Internet-delivered psychological treatments for mood and anxiety disorders: a systematic review of their efficacy, safety, and cost-effectiveness. PLoS One. 2014;9(5):e98118. doi: 10.1371/journal.pone.0098118. https://dx.plos.org/10.1371/journal.pone.0098118. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Hollinghurst S, Peters T, Kaur S, Wiles N, Lewis G, Kessler D. Cost-effectiveness of therapist-delivered online cognitive-behavioural therapy for depression: randomised controlled trial. Br J Psychiatry. 2010 Oct;197(4):297–304. doi: 10.1192/bjp.bp.109.073080. [DOI] [PubMed] [Google Scholar]
- 55.Kolovos S, Kenter RM, Bosmans JE, Beekman AT, Cuijpers P, Kok RN, van Straten A. Economic evaluation of internet-based problem-solving guided self-help treatment in comparison with enhanced usual care for depressed outpatients waiting for face-to-face treatment: A randomized controlled trial. J Affect Disord. 2016 Aug;200:284–92. doi: 10.1016/j.jad.2016.04.025. [DOI] [PubMed] [Google Scholar]
- 56.Phillips R, Schneider J, Molosankwe I, Leese M, Foroushani PS, Grime P, McCrone P, Morriss R, Thornicroft G. Randomized controlled trial of computerized cognitive behavioural therapy for depressive symptoms: effectiveness and costs of a workplace intervention. Psychol Med. 2013 Jun 24;44(4):741–752. doi: 10.1017/s0033291713001323. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Gerhards S, de Graaf L E, Jacobs L, Severens J, Huibers M, Arntz A, Riper H, Widdershoven G, Metsemakers J F M, Evers S M A A. Economic evaluation of online computerised cognitive-behavioural therapy without support for depression in primary care: randomised trial. Br J Psychiatry. 2010 Apr;196(4):310–8. doi: 10.1192/bjp.bp.109.065748. [DOI] [PubMed] [Google Scholar]
- 58.Longworth L, Rowen D. Mapping to obtain EQ-5D utility values for use in NICE health technology assessments. Value Health. 2013 Jan;16(1):202–10. doi: 10.1016/j.jval.2012.10.010. https://linkinghub.elsevier.com/retrieve/pii/S1098-3015(12)04161-7. [DOI] [PubMed] [Google Scholar]
- 59.Pickard A, Knight S. Proxy evaluation of health-related quality of life: a conceptual framework for understanding multiple proxy perspectives. Med Care. 2005 May;43(5):493–9. doi: 10.1097/01.mlr.0000160419.27642.a8. http://europepmc.org/abstract/MED/15838415. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Rush J, First MB, Blacker D. Handbook of Psychiatric Measures. USA: American Psychiatric Association; 2002. May, pp. 639–640. [Google Scholar]
- 61.National Collaborating Centre for Mental Health . Depression in Adults with a Chronic Physical Health Problem. Leicester, UK: National Collaborating Centre for Mental Health (UK); 2010. [Google Scholar]
- 62.Drummond M, Barbieri M, Cook J, Glick H, Lis J, Malik Farzana, Reed Shelby D, Rutten Frans, Sculpher Mark, Severens Johan. Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report. Value Health. 2009 Jun;12(4):409–18. doi: 10.1111/j.1524-4733.2008.00489.x. https://linkinghub.elsevier.com/retrieve/pii/S1098-3015(10)60782-6. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplementary file 1.