Abstract
myCOPD is a digital tool designed for people to manage their chronic obstructive pulmonary disease (COPD). It requires a device with an internet connection and incorporates tools for education, self-management, symptom tracking and pulmonary rehabilitation (PR). myCOPD was selected for medical technologies guidance by the UK National Institute for Health and Care Excellence (NICE) in 2020. The External Assessment Group (EAG) critiqued the company’s submission. The evidence comprised four clinical studies (three randomised controlled trials [RCTs] and one observational study) and real-world evidence from 22 documents. The RCTs had small sample sizes, limiting the power to detect statistically significant differences and to match patient characteristics across arms. The company produced two de novo models for two subgroups of people with COPD; people discharged from hospital with acute exacerbation of COPD (AECOPD) and people referred for PR. After the EAG updated input parameters and adjusted the model structures, cost savings of £86,297 per clinical commissioning group (CCG) compared with standard care were estimated for the AECOPD population, with myCOPD predicted to be cost saving in 74% of iterations. Cost savings of £22,779 per CCG were estimated for the PR population (with the assumption that the CCG had an existing myCOPD licence), with myCOPD predicted to be cost saving in 86% of the iterations. The Medical Technologies Advisory Committee concluded that although myCOPD has the potential to help manage COPD in adults, further evidence is required to address uncertainties in the current evidence base. NICE published this as Medical Technology Guidance 68 (National Institute for Health and Care Excellence (NICE). myCOPD for managing chronic obstructive pulmonary disease. 2022. Available at: https://www.nice.org.uk/guidance/mtg68/).
Supplementary Information
The online version contains supplementary material available at 10.1007/s40258-023-00811-x.
Key Points for Decision Makers
| myCOPD has the potential to help adults manage their chronic obstructive pulmonary disease (COPD); however, there is currently insufficient evidence to support the case for routine adoption in the NHS. |
| Research comparing myCOPD with standard care is recommended to address uncertainties about the claimed benefits of using myCOPD. |
Introduction
The National Institute for Health and Care Excellence (NICE) evaluates medical technologies and produces evidence-based guidance to help improve health and social care in England. NICE’s Medical Technologies Evaluation Programme (MTEP) selects medical technologies for evaluation that have the potential to reduce costs to the healthcare system and offer clinical benefits to the patient compared with standard care.
A scoping document detailing the decision problem is produced by NICE, and clinical and economic evidence submitted by the company is assessed independently by an External Assessment Group (EAG). Following the EAG’s evaluation and public consultation period, the Medical Technologies Advisory Committee (MTAC) develops guidance [2, 3].
In 2019, NHS England commissioned NICE to develop a medical technologies guidance development process for digital health technologies (DHTs), which is being piloted on a selected number of topics. The proposed process is based on the NICE MTEP process but is adapted to consider some of the characteristics of digital technologies.
In March 2022, NICE issued final guidance on myCOPD for managing chronic obstructive pulmonary disease (COPD). myCOPD is a digital tool designed to enable people to manage their COPD. The platform is designed to allow shared decision making between patient and clinician to improve self-efficacy and help individuals to manage their COPD effectively with the support of myCOPD.
The EAG critiquing the evidence was York Health Economics Consortium. Clinical experts, identified using NICE’s published processes, provided advice to the EAG and MTAC.
This article includes an overview of the clinical and cost-effectiveness evidence submitted by the company, the EAG’s report, and subsequent development of the NICE guidance. Full documentation including final guidance and supporting evidence is available on the NICE website [1].
Background
COPD is a lung disease characterised by airflow limitation [4]. It can be caused by exposure to harmful substances, for example cigarette smoking, and is a common cause of death worldwide [4]. COPD is common in adults and treatment involves strategies to control symptoms, improve quality of life and reduce exacerbations and mortality [4]. Pharmacological treatments include bronchodilators, inhaled corticosteroids, systemic glucocorticoids, phosphodiesterase-4 inhibitors and antibiotics. Non-pharmacological treatment includes smoking cessation and pulmonary rehabilitation (PR) [4]. myCOPD (my mHealth) is an application (app) that can be installed onto a device such as a smartphone, tablet or computer via an internet connection. Licencing is available via different price packages. At the time of the assessment, an ‘unlimited licence package’ was available as part of a 3-year contract with a clinical commissioning group (CCG). myCOPD incorporates tools for patient education, self-management, symptom tracking and PR. Individuals can record their symptoms daily and they may periodically undertake a COPD assessment test (CAT). With the user’s permission, clinicians can access the results of these assessments and patient medication records, so that monitoring and management (for example, suggesting a change to inhaler prescriptions) can be undertaken remotely.
The PR element of the tool is a 6-week online course comprising incremental exercise training and education sessions promoting effective self-management of COPD, and is conducted remotely.
Decision Problem (Scope)
Population
The population described in the scope included people with a diagnosis of COPD. The company did not propose any variation to the scope but in the evidence submission, the company focused on two subgroups: a cohort who had been discharged from hospital for acute exacerbation of COPD (AECOPD) and a cohort with stable COPD eligible for PR. It is likely that there is a small overlap between these populations.
Intervention and Comparator
The intervention identified in the scope was myCOPD as an add-on intervention to standard care, and the comparator listed in the scope was standard care without myCOPD.
All of the clinical studies identified by the EAG assessed myCOPD in addition to standard care and compared it with standard care alone; however, the definition of standard care varied across studies.
Outcomes
The scope included 11 outcomes. The company provided evidence for all of these, including COPD symptom assessment CAT score, rates of acute exacerbation and hospital admissions, number of healthcare professional (HCP) consultations (limited real-world evidence only), rates of inhaler error, adherence to the use of myCOPD, health-related quality of life (HRQoL), patient activation measures, self-efficacy for appropriate medication use, a walking test and device-related adverse events.
Review of Clinical and Economic Evidence
Clinical Effectiveness Evidence Identification and Selection
The company submitted four published peer-reviewed clinical studies, including three randomised controlled trials (RCTs) [5–7] and one observational study [8]. One study was ongoing at the time of its submission [9]. The company also submitted six published and nine unpublished real-world evaluations (RWEs) of myCOPD conducted in seven settings (Ipswich and East Suffolk [10], Southend [11], Leeds [12], Coventry [13], Grampian [14], Highland [15] and West Lothian [16]). The company shared usage information (as of January 2021) [17] and unpublished responses from six clinicians to a company questionnaire about the usefulness of the app and whether myCOPD could be incorporated into the clinical pathway [18].
The company did not provide information on the selection criteria or search strategy to identify relevant studies in its submission report. It was therefore not possible to assess whether the search methodology was appropriate, and hence the EAG conducted a de novo literature search (see the electronic supplementary material [ESM]) to identify evidence.
The EAG searched a range of resources containing details of published, unpublished and ongoing research. The search was originally conducted in October 2019, then updated in January 2021 after it was selected for the guidance development. From the 2019 and 2021 searches, 7761 records in total were retrieved, with 3280 remaining after deduplication for assessment.
The EAG’s search did not identify any clinical studies that were not stated in the company’s submission, but a further five published RWE papers were identified by the EAG’s searches, including an additional three RWE settings (Kent [19], Mid and South Essex [20] and Dorset [21]).
No meta-analysis was conducted by the company or by the EAG.
Critique of Clinical Effectiveness Evidence
Evidence on effectiveness was available for the two myCOPD applications: self-management and PR. For self-management, two RCTs (RESCUE [7] and EARLY [6]), one observational study [8] and a number of RWE reports provided evidence relating to the use of myCOPD. For PR, one RCT (TROOPER) [5] and a number of RWE reports provided evidence relating to the use of myCOPD. For the quality assessment of studies, the EAG used the criteria proposed by the Centre for Reviews and Dissemination for RCTs (Khan et al., 2001) and the Critical Appraisal Skills Programme (CASP) checklist for cohort studies for the comparative observational studies (CASP UK 2013).
The company did not conduct a critical appraisal of the included studies and the EAG undertook its own critical appraisal for each of the clinical studies. The internal and external validity of the RCTs were considered to be acceptable. For the observational study [8], external validity was judged to be acceptable but internal validity was judged to be low because of limited reporting of intervention details and outcome measures and no assessment of confounding factors. The EAG did not conduct quality assessment of real-world evidence because of the unsubstantive nature of outcomes and variation in the methods in these studies. The EAG considered these studies to have acceptable external validity but low internal validity.
Self-Management
The two RCTs evaluating myCOPD for self-management of COPD were RESCUE (compared myCOPD with ‘usual care with additional written support’) [7] and EARLY (compared myCOPD with 'usual care') [6]. Both trials were small, including 41 and 60 participants respectively. Details of power calculations were reported in EARLY. In RESCUE, the authors noted that the study was small and had limited power to demonstrate the effects on all measured outcomes. However, the authors considered that the sample size was suitable for the main objective of their pilot study—to determine the feasibility of using a digital platform to support participants with COPD after a significant clinical event such as exacerbation. Outcomes were measured at baseline and at three months in both trials. The trials did not have any longer-term follow up.
In terms of outcome for the RCTs, although there was a tendency for myCOPD to improve the CAT score in the RCTs, findings were not statistically significant and positive effects were not consistently shown for other outcomes (acute exacerbations, inhaler errors and HRQoL).
The observational study [8] explored the efficacy of myCOPD use compared with standard care without using myCOPD. The study matched the scope of the decision problem in terms of its populations; it was small (n = 36) and was reported as a brief article with limited details of the study methods.
RWE focused on user acceptance and adherence, and many were pilot studies using the results to inform decisions on whether to commission the app more widely.
Pulmonary Rehabilitation
The RCT of myCOPD for PR, TROOPER [5], is a non-inferiority RCT that compared myCOPD PR with a ‘face-to-face’ PR programme. There were 90 participants in the trial and although details of the power calculation were reported, there was uncertainty regarding its validity. Outcomes were measured at baseline and one week after completion of a six-week PR programme in TROOPER with no longer-term follow up.
The TROOPER trial found that myCOPD was non-inferior compared with face-to-face PR for CAT score (adjusted mean difference −1.0, 95% confidence interval [CI] −2.9 to 0.86; p = 0.373) and six-minute walking test (adjusted mean difference 23.8 m, 95% CI −4.5 to 52.2; p = 0.098), and there was an indication of non-inferiority for other measures, e.g. anxiety and quality of life. Rates of acute exacerbation and hospital admissions were not reported; however, due to the small sample size in this trial, caution was noted in the interpretation of findings.
The RWE for PR focused on user acceptance and adherence and many were pilot studies using the results to inform decisions on whether to commission the app more widely.
Clinical Effectiveness Summary
Overall, the evidence was considered uncertain due to the non-significant findings in the RCTs of self-management and the lack of power in the RCT of PR. Positive findings from RWE suggested the potential for real-world implementation but these studies are mainly interim service evaluations that do not provide robust evidence of effectiveness. App usage fell over time in all three RCTs and in the RWE, and there was some question over the long-term effectiveness.
Economic Evidence
The economic evidence was assessed in July 2021. The company did not include any economic evidence in its submission. The EAG rerun the searches based on the company search methods, as well as conducting a de novo economic evidence literature search (see the ESM). No economic studies suitable for inclusion were identified by the EAG.
The company submitted two de novo cost models comparing myCOPD with standard care. The company focused their models on subgroups of the COPD population in line with the clinical evidence.
Acute Exacerbation of Chronic Obstructive Pulmonary Disease Model (Self-Management)
The first subgroup modelled was a cohort who had been discharged from hospital for AECOPD. Standard care for this population was a written self-management plan at discharge. A cost calculator with a one-year time horizon was developed in Treeage. The model was based on a CCG purchasing the unlimited myCOPD licence package (myCOPD was covered for the whole CCG). Outcomes were modelled over three months for myCOPD and standard care, and included the rate of general practioner (GP) appointments, non-hospital admitted exacerbations and hospital readmissions for COPD (Table 1). Efficacy data for the model, including the rate of exacerbations and hospital readmissions, were sourced from the RESCUE RCT [7], and the number of GP appointments was sourced from the RWE by McLaughlin and Skinner [14] (see Table 1).
Table 1.
Clinical parameters used in the company’s model and any changes made by the EAG [1]
| Parameter | Company value | EAG value | EAG comment |
|---|---|---|---|
| Model 1 AECOPD (self-management) | |||
| Mean number of patients registered in a CCG | 447,464 | No change | QOF 2019/20 [27]. Used to calculate the annual licence fee |
| Average number of admissions for AECOPD per 100,000 in England | 247 | No change | PHE Inhale. All of England [28]. Used to calculate the number of admissions for AECOPD in an average CCG |
| Uptake of myCOPD | – | 46% | RESCUE study [7]. 124 people eligible and 67 people did not go on to use myCOPD |
| Number of exacerbations over 90 days post AECOPD (SOC arm) | 1.88 | No change | RESCUE study [7]. Used to calculate the overall cost of non-admitted exacerbations |
| Number of exacerbations over 90 days post AECOPD (myCOPD) | 1.06 | 1.09 | RESCUE study [7]. Used to calculate the overall cost of non-admitted exacerbations. The EAG calculated this using the adjusted rates ratio (0.581) rather than the raw risk (SOC exacerbation of 1.88*0.581 = 1.09) |
| Number of GP appointments over 90 days post AECOPD (SOC arm) | 2.28 | No change | McLaughlin and Skinner 2020 [14]. Used to calculate the overall cost of GP appointments |
| Number of GP appointments over 90 days post AECOPD (myCOPD) | 1.85 | No change | McLaughlin and Skinner 2020 [14]. Used to calculate the overall cost of GP appointments |
| Readmission rate over 90 days post AECOPD (SOC arm) | 0.39 | No change | RESCUE study [7]. Table 5. Used to calculate the overall cost of readmissions |
| Readmission rate over 90 days post AECOPD (myCOPD) | 0.24 | 0.20 | RESCUE study [7]. Table 5. Used to calculate the overall cost of readmissions. The EAG converted the 3-month adjusted odds ratio (0.383) to a relative risk and multiplied it by the readmission rate seen in the SOC arm (0.39) |
| Model 2 PR | |||
| Probability of having a diagnosis of COPD in the general population | 1.94% | No change | NHS Digital 2020 [27, 29]. Used to calculate the number of patients with COPD, which contributes to calculation of the number of patients entering the model |
| Proportion of patients eligible for PR referral | 29.7% | No change | NHS Digital 2020 [27]. Average of patients with COPD and MRC ≥3 (denominator plus PCAs) divided by the average of COPD patients per CCG. Used to calculate the number of patients eligible for a PR referral, which contributes to the calculation of the number of patients entering the model |
| Proportion of eligible patients referred for PR (SOC) | 20.2% | No change | Company model: 40% eligibility rate from COPD prime applied to QOF data. 15% of these patients were assumed to be offered PR based on COPD Prime [30]. The resulting number is then applied to the QOF data eligible patients to calculate the percentage. Used to calculate the number of patients referred for PR, which contributes to the calculation of the number of patients entering the model |
| Proportion of eligible patients referred for PR (myCOPD) | 20.2% | No change | Assumed to be the same capacity when myCOPD is introduced. Used to calculate the number of patients referred for PR, which contributes to the calculation of number of patients entering the model |
| Median patients referred to PR service (PR service costing scenario only) | 495 | No change | Median of 298 reported per CCG in NACAP 2019 [31] over 6 month period (multiplied by 2 to give yearly referral rate). 84% of referrals are reported to be for COPD in NACAP 2018 [32]. Used to calculate the number of patients entering the model for the PR service costing scenario |
| Probability of being treated with hybrid | 11% | 12% | Assumption that uptake will be similar to that of myCOPD alone. Recalculated by the EAG to account for both people who completed and did not complete the courses. Note that uptake is different between the AECOPD and the PR model because the patient populations in the models are different and therefore it is expected their willingness to use myCOPD may be different |
| Probability of being treated with myCOPD only | 11% | 12% | Based on the proportion of patients who took up remote PR in the Southend study [33]. See the above comment. It is also noted that no hybrid approach was offered in the study and therefore this assumes that uptake of myCOPD alone would be unchanged if a hybrid approach was also offered |
| Number of patients entering the model (both arms) | 2,577 | 127 | Calculated based on all patients eligible for PR referral. Decision point in the model changed so that only those who are willing to use myCOPD enter the model, because the introduction of myCOPD is only expected to influence costs and outcomes for these patients |
| Number of patients entering the model (PR service costing scenario) | 495 | 121 | Calculated based on all patients referred to PR services. Decision point in the model changed so that only those who are willing to use myCOPD enter the model |
| Probability of starting and completing a PR course (face-to-face PR) | 41.9% | No change | COPD Prime reports of those referred, 59% start a PR course and 71% complete [30]. Used in the model to determine the completion rate for face-to-face PR |
| Probability of starting and completing a PR course (hybrid PR) | 41.9% | No change | Assumed equal to face-to-face PR based on the TROOPER study [5]. Used in the model to determine the completion rate for hybrid PR |
| Probability of starting and completing a PR course (myCOPD only PR) | 41.9% | No change | Assumed equal to face-to-face PR based on the TROOPER study [5]. Used in the model to determine the completion rate for myCOPD PR. The EAG noted that completion for the myCOPD arm in the trial was only 62% (compared with 72% for face-to-face PR). However, non-inferiority was still demonstrated and patients were asked to do five sessions per week in the myCOPD arm as opposed to two sessions in the face-to-face group |
| Annual exacerbations after completing PR | 2.11 | No change | Used in the model to calculate exacerbations following completion of any PR programme [30] |
| Annual exacerbations after not completing PR | 3.31 | No change | Used in the model to calculate exacerbations following non-completion of any PR programme [30] |
| Waiting time for PR after assessment (days) | 13 | No change | NACAP 2020 [31]. Used to inform the cost of waiting for those having face-to-face PR |
AECOPD acute exacerbations of chronic obstructive pulmonary disease, CCG clinical commissioning group, COPD chronic obstructive pulmonary disease, EAG External Assessment Group, GP general practitioner, MRC Medical Research Council, NACAP National Asthma and Chronic Obstructive Pulmonary Disease Audit Programme, NHS National Health Service, PCAs personalised care adjustments, PR pulmonary rehabilitation, QOF Quality and Outcomes Framework, SOC standard of care
Base-case results from the company’s AECOPD model estimated that myCOPD generated cost savings of £204,641 per CCG. Best- and worst-case scenarios examined the impact of varying the model inputs on the results (best-case scenario: £1,785,878 cost saving per CCG; worst-case scenario: £69,530 cost increase per CCG; see ESM Table S3 for inputs). A tornado diagram showed that the readmission rate over 90 days post AECOPD was the key driver of results. The 90-day readmission rate in the myCOPD arm at which the base-case model became cost neutral/cost incurring was 0.357.
Overall, the EAG considered the company’s AECOPD model structure to be appropriate. However, the model outcomes were applied to every person who was discharged from hospital with AECOPD, meaning the uptake of myCOPD was assumed to be 100%. The EAG considered the 100% uptake rate to be optimistic and amended the uptake rate to 46% to account for the proportion of people who would not agree to be registered for myCOPD. This was based on data from an RCT [22]. The uptake rate was varied in a sensitivity analysis to account for any uncertainty in this value. No other changes were made to the model structure but existing inputs were updated (see Tables 1 and 2).
Table 2.
Cost parameters used in the company’s model and any changes made by the EAG [1]
| Parameter | Company | EAG | Source |
|---|---|---|---|
| Model 1 AECOPD | |||
| Technology costs (per registered patient) | £0.25 pa | No change | Company submission |
| Exacerbation self-managed or managed in primary care | £53.59 | £81.75 | Adapted from Jordan et al., 2015 [34]. EAG updated with 2019/2020 reference costs [35] |
| Emergency hospital admission for AECOPD | £1583 | £1721 | COPD PRIME [30] (updated with reference costs 2018/2019 [36]). EAG updated with 2019/2020 reference costs [35] |
| GP appointment (9.2 mins) | £39 | No change | PSSRU 2020 [37] |
| Practice nurse per hour (band 5) to register patients and train | £39 | No change | PSSRU 2020 [37] |
| Practice manager to administer top-level licences | £48 | No change | PSSRU 2020 [37] |
| CCG licence set-up | £360 | No change | 7.5 h of a practice manager’s time at £48 an hour (PSSRU 2020 [37] and company assumption) |
| Training of one clinician per CCG to use myCOPD | £1950 | No change | QOF data give an average of 50 GP practices per CCG. Company: PSSRU 2020 [37] Training given by a band 5 practice nurse [36, 37] |
| Patient myCOPD licence registration (per year) | £9.75 | £19.50 | Company submission: 15 min to register a patient by a band 5 practice nurse [37]. Clinical experts queried by the EAG gave a range of between 15 and 45 min, and potentially by band 6 or 7 staff. The EAG judged it appropriate to be conservative and assume 30 min for the base-case value, with a range of 15 (band 5) to 45 min (band 6) used in sensitivity analysis |
| Model 2 PR | |||
| Annual cost of myCOPD per patient (PR service costing scenario only) | £10,000 | No change | Provided by the company |
| Cost to administer licences (PR service costing scenario only) | £360 | No change | Practice manager assumed to administer top-level licences at a cost of £48 per hour [37]). Assumed to take 1 day |
| Cost of training for a PR service to use myCOPD (PR service costing scenario only) | £195 | No change | Assumed to be five band 5 staff trained for 1 h each to reflect the PR service being delivered more centrally. Costed using PSSRU 2020 [37] |
| Face-to-face PR programme | £695 | No change | COPD PRIME (updated using PSSRU 2020 staff costs [37]). Assumed to include the cost of initial and post-discharge assessment |
| Face-to-face PR assessment | £79 | No change | PSSRU 2020 [37], expert opinion – 1 h of band 6 and 1 h of band 4. The same cost was applied for initial assessment and post-discharge assessment |
| Cost per exacerbation | £283 | £328 | 15% probability of exacerbation being treated in hospital (COPD PRIME [30]) multiplied by the cost of a hospital admission for exacerbation. 85% probability of exacerbation being treated in primary care multiplied by the cost of a non-admitted exacerbation. Costs of admitted and non-admitted exacerbation as per the AECOPD model |
| Telephone support for remote PR (myCOPD only) | £18 | No change | Expert opinion three 10-min phone calls. Assumed to be a band 6 community therapist at a cost of £49 per hour (PSSRU 2020 [37]) |
| Cost to register a patient for a myCOPD licence | £9.75 | £19.50 | Company submission assumed 15 min of band 5 practice nurse time at a cost of £39 per hour [37]). Clinical experts queried by the EAG gave a range of between 15 and 45 min, and potentially by band 6 or 7 staff. The EAG judged it appropriate to be conservative and assume 30 min for the base-case value, with a range of 15 (band 5) to 45 min (band 6) used in sensitivity analysis |
| Cost of time waiting for assessment | £33 | £39 | Company: Cost per exacerbation (as above) multiplied by a total waiting time of 13 days multiplied by the annual number of exacerbations in people who did not complete PR (3.31) |
| Cost of starting and not finishing PR: face-to-face PR | – | £26 | Assumed to be the cost of face-to-face PR minus assessment costs, divided by 6. Reflects the cost of patients attending one session before dropping out. Applied to 29% of patients (based on the COPD PRIME tool, which states 59% of those referred start PR, and of those starting, 71% complete their PR programme [37]) |
| Cost of starting and not finishing PR: Hybrid | – | £13 | Assumed to be the cost of starting and not finishing face-to-face PR halved |
| Cost of starting and not finishing PR: myCOPD | – | £2 |
Assumed to be the cost of one support phone call [37] Proportion starting but not finishing PR was assumed to be the same as face-to-face PR |
AECOPD acute exacerbations of chronic obstructive pulmonary disease, CCG clinical commissioning group, COPD chronic obstructive pulmonary disease, EAG External Assessment Group, GP general practitioner, PR pulmonary rehabilitation, PSSRU Personal Social Services Research Unit, QOF quality and outcomes framework
The changes made to the AECOPD model by the EAG resulted in a reduced cost saving, from £204,641 to £86,297 per CCG, influenced mainly by the amended uptake in the model, although reducing the proportion of AECOPD patients who registered for myCOPD also reduced resource use costs.
In the EAG’s model, the best-case and worst-case scenario results were a cost saving of £4,143,428 per CCG (best-case scenario) and a cost increase of £58,928 per CCG (worst-case scenario). The EAG also identified the key driver from the sensitivity analysis as the hospital 90-day readmission rate. The point at which myCOPD changed from being cost saving to cost incurring was when the uptake rate was 26.2% or when the per person 90-day readmission rate was 0.30.
The EAG conducted probabilistic sensitivity analysis (PSA), which estimated that myCOPD had a 73.5% probability of being cost saving (see ESM Table S4 for sensitivity analysis inputs).
Pulmonary Rehabilitation Model
The second subgroup modelled was a cohort of patients eligible for PR with stable COPD, defined as Medical Research Council (MRC) grade three dyspnoea or above, who had been discharged from hospital post AECOPD. Standard care for this population was face-to-face PR (a six-week programme consisting of two supervised and three additional unsupervised sessions a week).
The company developed a decision tree in Treeage. Patients enter the model with a face-to-face PR assessment. Patients in the myCOPD arm can either be treated with face-to-face PR, a mixture of face-to-face sessions plus use of myCOPD, or use of myCOPD only. The PR element of the myCOPD app is intended to be used instead of standard care, deviating from the NICE scope, which specifies for myCOPD to be used in addition to standard care. Patients did not have to complete the full PR programme. Outcomes were modelled over a one-year time horizon and included the rate of exacerbations (see Table 1 for parameter inputs). The company presented two sets of results—incremental costs per CCG and incremental costs per PR provider. It must be noted that the former set of results cannot be interpreted as stand-alone because it was assumed that the myCOPD licencing is already in place.
Base-case results from the company’s PR model estimated that myCOPD generated cost savings of £20,269 per CCG; the alternative scenario presented a cost saving of £8707 per PR service provider. The company presented a tornado diagram to present the key drivers for the CCG costing model results (see ESM Table S5 for inputs). For the base case, the only parameter that changed the direction of the results was the number of referrals to PR when using myCOPD. For the PR service scenario, the key drivers identified by the company were the probability of being treated with myCOPD only and with a hybrid model (face-to-face and myCOPD), and the cost per face-to-face PR treatment. The company also presented a threshold analysis that identified that 276 referrals per year to PR were necessary for myCOPD to be cost saving.
Overall, the company’s PR model and structure were judged to be appropriate. However, the EAG concluded that a cost should be included for people who begin, but do not complete, the PR programme. The EAG included this, but any benefits of partially completing a PR programme were not captured in the number of exacerbations assigned to patients not completing a PR programme.
The EAG also changed the decision point in the model from referral to PR, to the point at which patients have expressed their willingness to use myCOPD. This was to better align the starting point of the two myCOPD models, to align the treatment arm with the scope (treatment options being myCOPD alone or hybrid option only, removing the option for only face-to-face), and to align the model with the population in the TROOPER study (the main efficacy data used in the model).
Changes made by the EAG to the PR model resulted in an increased cost saving, from £20,269 to £22,779 per CCG, or £8707 to £11,093 per PR service provider if considering the PR costing scenario.
Sensitivity analysis showed that for the base-case results, key drivers were the probability of being treated with myCOPD only, probability of being treated with the hybrid model, and the probability of referral to PR. In the base-case model, it is not possible for the results to become cost incurring unless the cost of registering a patient on the myCOPD app outweighs all of the other cost benefits. This is because the CCG is assumed to have already purchased myCOPD. In threshold analysis, myCOPD may not be cost saving in PR services with fewer than 240 referrals per year. Furthermore, the probability of being treated with myCOPD would need to be 1.9% when the hybrid model uptake is 12.2%, or 9.8% if the hybrid model is not used, for myCOPD to be cost saving. Therefore, if a hybrid model is not used, uptake of myCOPD would need to be higher to demonstrate a cost saving.
The EAG conducted PSA, which estimated myCOPD had an 86% probability of being cost saving in the CCG model and 87% probability of being cost saving in the PR service scenario.
Economic Effectiveness Summary
Due to no evidence of benefit in the whole population of people with COPD, the company modelled subgroups where benefit could be demonstrated (a cohort who had been discharged from hospital for AECOPD, and a cohort with stable COPD referred for PR).
The AECOPD model demonstrated that myCOPD was cost saving in this subpopulation, dependent on myCOPD uptake and hospital readmission rates. The PR model demonstrated additional CCG cost savings if myCOPD is used for the delivery of PR services, but the results of both models should not be combined due to overlap between patient populations. The PR costing scenario also demonstrated that purchasing a PR service licence for use of myCOPD exclusively for PR delivery is likely to be cost saving, provided sufficient uptake and annual referrals.
National Institute for Health and Care Excellence Guidance
Draft Recommendations
The evidence submitted by the company and the EAG’s critique of the evidence was presented to the MTAC, who considered draft recommendations relating to myCOPD following their meeting in September 2021. These were as follows:
myCOPD shows promise for self-managing COPD; however, there is not enough good-quality evidence to support the case for routine adoption.
Further comparative research is recommended to address uncertainties about myCOPD’s clinical benefits on outcomes, such as the rate of acute exacerbation and how it affects healthcare resource use such as hospital admissions.
Consultation Response
During the consultation, NICE received 108 consultation comments from 11 consultees (two company representatives, two HCPs, six members of the public and one professional organisation representative). These are collated into themes and are detailed in Table 3. Further detail can be found on the NICE MTG68 webpage [1]. The comments covered the population groups, clinical evidence, economic inputs and model uncertainty, inequality, the technology, myCOPD engagement and wording clarifications. The draft recommendation did not change after consultation.
Table 3.
Consultation comments summary [1]
| Theme | Detail |
|---|---|
| Population groups | Recommendations did not address the use of myCOPD specifically for the two modelled population groups: (1) people discharged from hospital following an acute exacerbation of COPD; and (2) people eligible for pulmonary rehabilitation |
| Powering of the TROOPER trial | The TROOPER trial of myCOPD PR versus face-to-face PR was adequately powered to show non-inferiority |
| Patient engagement | In the EAG report, the completion rate in the TROOPER trial was presented in a misleading way and that the concept of effective engagement had been ignored |
| Description of benefits | The benefits of myCOPD in trials and RWE were not properly acknowledged for the populations modelled |
| Economic model inputs | The uptake rate of myCOPD in the AECOPD model was thought to be inaccurate due to the RESCUE trial figure representing uptake to a clinical trial rather than the app itself. The EAG used a 46% uptake rate (reported in the RESCUE trial) [7]; a 48% uptake rate was suggested. Staff time in the economic models to represent registering people for myCOPD was too long and it was suggested it be reduced |
| Uncertainty in the cost modelling | The guidance appeared to overstate the effect of the uncertainties on whether myCOPD is cost saving or not. The EAG conducted sensitivity analysis to present this more clearly |
| Further research | There were several comments questioning why further research was necessary if the economic models showed a cost saving. It was unclear what research was needed |
| Patient-related considerations | It was stated that the patient expert was not representative of users. There was a lack of representation of NHS clinicians with sufficient experience of using myCOPD, particularly relating to PR. Only one of three invited clinicians were able to attend the initial MTAC meeting |
| Technology | Comments regarded updates to the app, accuracy of the data input and security of the data. The company responded to note all clinical information is automatically updated, and the app detects any possible error data input and meets all applicable security standards |
| Integration of myCOPD | There were questions around the integration of myCOPD with NHS systems. Healthcare professionals are able to track the use of myCOPD |
| Engagement | The consultee believed that the committee misunderstood adherence in the context of effective engagement and intervention. The company provided references to show how engagement is linked to behavioural change [23, 25] and agreed that understanding why people stop using the app could be an area of future research. Another comment questioned whether varying levels of engagement due to ill health could skew the data |
| Equalities | A healthcare professional queried if the use of myCOPD would increase health inequalities because of digital literacy and access to smart devices. Some people may need extra support to use the app. The company agreed further research is needed |
| Wording clarifications | Additional wording to be added regarding the care pathway description to ensure PR is covered. Wording changes around the technology and use were suggested |
AECOPD acute exacerbations of chronic obstructive pulmonary disease, app application, COPD chronic obstructive pulmonary disease, EAG External Assessment Group, MTAC Medical Technologies Advisory Committee, NHS National Health Service, PR pulmonary rehabilitation, RWE real-world evaluations
Key Challenges and Learning Points
One challenge the EAG and committee came across was understanding the uptake of myCOPD due to the paucity of evidence in the specific populations being modelled. Uptake is of particular importance due to a subscription pricing model. At the time of submission, the cost of myCOPD was applied to the whole CCG population (rather than only individuals with COPD) and therefore the higher the uptake, the more cost saving myCOPD is shown to be. Since the submission, CCGs have ceased to exist, with integrated care systems taking over in July 2022. It is unknown how the pricing scheme currently works and how this will change the licensing system and model results.
A challenge of evaluating patient-facing DHTs is understanding how the level of engagement with an application (such as myCOPD) links to a change in clinical outcomes. Outcomes from the evidence used in the economic models were short term (e.g. three months in the RESCUE study), which makes it difficult to know if the observed benefits would continue with ongoing use of the app or whether they would wane, or alternatively, whether continued engagement is necessary. Yardley et al. emphasised the importance of ‘effective engagement’, which is defined as the engagement necessary to produce the desired outcomes [23]. There are several ways of measuring engagement (rather than the ‘more engagement, the more effective’ approach) and a framework has recently been developed to help link usage to clinical outcomes in DHTs [23–25]. These could be used when designing studies for the effectiveness of DHTs, such as myCOPD. Furthermore, clinical experts have suggested that support from the health service may improve patient engagement [1].
Another challenge was the small sample size of the included RCTs. For the two trials evaluating the use of myCOPD for self-management (RESCUE and EARLY), no positive significant benefits were shown for any of the clinical outcomes: CAT score, acute exacerbations or hospital admissions, and there was no significant difference in inhaler errors or HRQoL; however, the sample size in these studies was small (n = 41 and 60, respectively). No power calculation was reported for the RESCUE trials, but the authors noted that the study had limited power to demonstrate effects on all measured outcomes. For the EARLY trial, a power calculation indicated that the trial was suitably powered; however, the authors noted that the small sample size and baseline differences between groups limited their ability to demonstrate statistically significant differences. From these trials, it was therefore unclear whether myCOPD had benefits for the self-management of COPD.
For the trial evaluating the use of myCOPD in PR compared with face-to-face PR (TROOPER), it was reported that the trial was adequately powered and the study showed that myCOPD PR was equivalent to face-to-face PR. The EAG assessed the trial power calculation and determined that the distance used to determine a minimal clinically important difference in the six-minute walking test (40.5 m) was higher than the distance in some recommendations (30 m) [26], resulting in an underestimation of the sample size needed to prove non-inferiority. In addition, the sample size had been calculated based on 58% power, lower than commonly used rates of 80–90%. It was therefore assessed that despite claims for suitable power, the study was likely to be too small to conclusively determine whether myCOPD in PR was equivalent to face-to-face PR.
Conclusions
The MTEP evaluation process was followed for the development of medical technologies guidance on myCOPD. This included a submission of clinical and economic evidence by the company, critical appraisal of this evidence by the EAG, drafting of recommendations by the MTAC, and a subsequent consultation. Following this process, the MTAC judged that myCOPD has the potential to help self-manage COPD in adults, but uncertainties in the evidence first need to be addressed [1].
Supplementary Information
Below is the link to the electronic supplementary material.
Acknowledgements
The authors are grateful to the clinical experts identified by NICE, who provided expertise and clinical knowledge, and also to Michelle Green, part of the EAG, and Catherine Bowe, who proofread the manuscript. This article was not externally peer reviewed by Applied Health Economics and Health Policy.
Declarations
Funding
York Health Economics Consortium is funded by NICE to act as an EAG for the MTEP.
Conflicts of interest
This summary of the Medical Technology Guidance was produced following publication of the final guidance report. Heather Davies, Judith Shore, Mary Chappell, Mick Arber, Angaja Phalguni and Stephanie Wake work or worked for the EAG, but otherwise have no conflicts of interest to declare. Yingying Wang is a employee of NICE and had no role in the production of the assessment report but reviewed and approved this manuscript.
Availability of data and material
The final guidance report MTG68 (https://www.nice.org.uk/guidance/mtg68) includes full details of the guidance.
Ethics approval
Not applicable.
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Code availability
Not applicable.
Author Contributions
This manuscript was prepared by HD and MC with contributions from JS and MA. Literature searching was undertaken by MA; the clinical evidence review and synthesis was undertaken by AP and SW; and the model critique was undertaken by JS and HD. MC responded to consultation comments. All authors reviewed the manuscript. Yingying Wang is an employee of NICE and had no role in the production of the assessment report but reviewed and approved this manuscript.
References
- 1.National Institute for Health and Care Excellence (NICE). myCOPD for managing chronic obstructive pulmonary disease. 2022. Available at: https://www.nice.org.uk/guidance/mtg68/.
- 2.National Institute for Health and Care Excellence (NICE). Medical technologies evaluation programme methods guide. 2017. Available at: https://www.nice.org.uk/process/pmg33/chapter/introduction. [PubMed]
- 3.National Institute for Health and Care Excellence (NICE). Medical technologies evaluation programme process guide. 2017. Available at: https://www.nice.org.uk/process/pmg34/chapter/introduction. [PubMed]
- 4.Agarwal AK, Raja A, Brown BD. Chronic obstructive pulmonary disease. Treasure Island: ©2022, StatPearls Publishing LLC; 2022. [PubMed]
- 5.Bourne S, DeVos R, North M, Chauhan A, Green B, Brown T, et al. Online versus face-to-face pulmonary rehabilitation for patients with chronic obstructive pulmonary disease: randomised controlled trial. BMJ Open. 2017;7(7):e014580. doi: 10.1136/bmjopen-2016-80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Crooks MG, Elkes J, Storrar W, Roy K, North M, Blythin A, et al. Evidence generation for the clinical impact of myCOPD in patients with mild, moderate and newly diagnosed COPD: a randomised controlled trial. ERJ Open Res. 2020;6:4. doi: 10.1183/23120541.00460-2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.North M, Bourne S, Green B, Chauhan AJ, Brown T, Winter J, et al. A randomised controlled feasibility trial of E-health application supported care vs usual care after exacerbation of COPD: the RESCUE trial. NPJ Digit Med. 2020;3:145. doi: 10.1038/s41746-020-00347-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.North M, Wilkinson T, Bourne S. The impact of an electronic self-management system for patients with COPD. Eur Respir J. 2014;44(Suppl 58):1413. [Google Scholar]
- 9.Chmiel F, Burns D, Pickering J, Blythin A, Wilkinson T, Boniface M. Retrospective development and evaluation of prognostic models for exacerbation event prediction in patients with chronic obstructive pulmonary disease using data self-reported to a digital health application [Preprint]. medRxiv; 2020.
- 10.Ipswich and East Suffolk CCG. Ipswich and East Suffolk Staff MyCOPD Feedback [unpublished data provided by my mHealth]. Ipswich and East Suffolk CCG; 2019.
- 11.Southend University Hospital. Study: 3 armed study into the use of myCOPD in pulmonary rehabilitation (Southend CCG). my mhealth Ltd; 2021. Available at: https://mymhealth-email-resources.s3.eu-west-2.amazonaws.com/lms/Southend+CCG.pdf.
- 12.Benn R. MyCOPD in Leeds (myCOPD Leeds evaluation). Leeds: 100% Digital Leeds; 2021. Available at: https://leedsdigitalinclusion.wordpress.com/our-work/key-initiatives/mycopd.
- 13.Heritage L. Use of the my mHealth myCOPD platform by the Coventry Community COPD Service during the Covid-19 pandemic (Coventry community project evaluation) [unpublished data]. Coventry: Coventry Community COPD Service; October 2020.
- 14.McLaughlin K, Skinner E. Evaluation MMH-E04. A real-world service evaluation of myCOPD. Results of a supported self-management evaluation of COPD patients by Stonehaven Medical Group and Aberdeenshire Health and Social Care Partnership (NHS Grampian). my mhealth Ltd; 2020. Available at: https://mymhealth.com/studies/real-world-service-evaluation-mycopd-mmh-e04.
- 15.Cooper R, Colligan J, Hamilton S, Finlayson E, Duffy M, Gilliatt J, et al. P150 Evaluation of myCOPD, a digital self-management technology for people with COPD, in a remote and rural population (NHS Highland) Thorax. 2021;76(Suppl 1):A169–A170. doi: 10.1136/thorax-2020-BTSabstracts.295. [DOI] [Google Scholar]
- 16.NHS Lothian. Project evaluation (NHS West Lothian) [unpublished data]. NHS Lothian; 2018
- 17.my mhealth Ltd. MyCOPD supplementary information [unpublished data]. My mhealth Ltd; 2021
- 18.my mhealth Ltd. My mhealth Clinician Feedback Questionnaire Responses 2020: myCOPD [unpublished data] × 6 responses. my mhealth Ltd; 2020.
- 19.Stokes J, Savage K. A digitally enhanced pulmonary rehabilitation approach for COPD during COVID-19 using myCOPD [conference abstract] Eur Respir J. 2021;58:2. [Google Scholar]
- 20.Mid and South Essex STP. myCOPD Mid and South Essex Case Study (myCOPD Mid and South Essex evaluation). my mhealth Ltd; 2021. Available at: https://mymhealth-email-resources.s3.eu-west-2.amazonaws.com/pdfs/mycopd_ms_essex_case_study.pdf.
- 21.Matheson-Monet C. Independent evaluation of the piloting of the implementation of my mhealth (myCOPD, myHeart and myDiabetes) by Dorset CCG (Dorset CCG evaluation). Chilworth, Hampshire: Wessex Academic Health Science Network; 2019. Available at: https://wessexahsn.org.uk/innovation-insight-library/113/independent-evaluation-of-the-piloting-of-the-implementation-of-mymhealth-mycopd-myheart-and-mydiabetes-by-dorset-clinical-commissioning-group.
- 22.Office for National Statistics. Exploring the UK’s digital divide. 2021. Available at: https://www.ons.gov.uk/peoplepopulationandcommunity/householdcharacteristics/homeinternetandsocialmediausage/articles/exploringtheuksdigitaldivide/2019-03-04
- 23.Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. 2016;51(5):833–842. doi: 10.1016/j.amepre.2016.06.015. [DOI] [PubMed] [Google Scholar]
- 24.Baumel A. Therapeutic activities as a link between program usage and clinical outcomes in digital mental health interventions: a proposed research framework. J Technol Behav Sci. 2022;7:234–239. doi: 10.1007/s41347-022-00245-7. [DOI] [Google Scholar]
- 25.Ainsworth B, Steele M, Stuart B, Joseph J, Miller S, Morrison L, et al. Using an analysis of behavior change to inform effective digital intervention design: how did the PRIMIT website change hand hygiene behavior across 8993 users? Ann Behav Med. 2017;51(3):423–431. doi: 10.1007/s12160-016-9866-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Holland AE, Spruit MA, Troosters T, Puhan MA, Pepin V, Saey D, et al. An official European Respiratory Society/American Thoracic Society technical standard: field walking tests in chronic respiratory disease. Eur Respir J. 2014;44(6):1428–1446. doi: 10.1183/09031936.0150314. [DOI] [PubMed] [Google Scholar]
- 27.NHS Digital. Quality and Outcomes Framework. 2020. Available at: https://digital.nhs.uk/data-and-information/publications/statistical/quality-and-outcomes-framework-achievement-prevalence-and-exceptions-data.
- 28.Public Health England. Inhale—INteractive Health Atlas of Lung conditions in England. 2019. Available at: https://fingertips.phe.org.uk/profile/inhale/data#page/3/gid/8000003/pat/46/par/E39000018/ati/165/are/E38000004/iid/93577/age/1/sex/4/cid/4/tbm/1/page-options/car-do-0_car-ao-1.
- 29.NHS Digital. Table 10: Prevalence, achievement and personalised care adjustments, respiratory group, COPD. NHS Digital; 2020.
- 30.Chartered Society of Physiotherapy. COPD PRIME tool. 2017. Available at: https://www.csp.org.uk/documents/copd-prime-tool.
- 31.National Asthma and Chronic Obstructive Pulmonary Disease Audit Programme (NACAP). Pulmonary Rehabilitation Organisational Audit 2019. 2020. Available at: https://www.rcplondon.ac.uk/projects/outputs/national-asthma-and-copd-audit-programme-nacap-pulmonary-rehabilitation-clinical-0.
- 32.National COPD Audit Programme. Pulmonary rehabilitation: an exercise in improvement. Royal College of Physicians; 2018. Available at: https://www.rcplondon.ac.uk/projects/outputs/pulmonary-rehabilitation-exercise-improvement-combined-clinical-and-organisational.
- 33.my mhealth Ltd. my mhealth Webinar: Digitally Enhanced Pulmonary Rehabilitation. My mhealth Ltd; 2021.
- 34.Jordan RE, Majothi S, Heneghan NR, Blissett DB, Riley RD, Sitch AJ, et al. Supported self-management for patients with moderate to severe chronic obstructive pulmonary disease (COPD): an evidence synthesis and economic analysis. Health Technol Assess. 2015;19(36):1–516. doi: 10.3310/hta19360. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.National Health Service. National Schedule of NHS Costs 2019/20. 2021. Available at: https://www.england.nhs.uk/national-cost-collection/.
- 36.National Health Service. National Schedule of NHS Costs 2018/19. 2020. Available at: https://www.england.nhs.uk/national-cost-collection/.
- 37.Curtis L, Burns A. Unit Costs of Health and Social Care 2020. Canterbury: 2020. Available at: https://www.pssru.ac.uk/project-pages/unit-costs/unit-costs-2020/.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.