Abstract
Background
Little is known about which self-management behaviors have the highest potential to influence exacerbation impact in COPD patients. We aimed to reach expert consensus on the most relevant set of self-management behaviors that can be targeted and influenced to maximize reduction of exacerbation impact.
Materials and methods
A 2-round Delphi study was performed using online surveys to rate the relevance and feasibility of predetermined self-management behaviors identified by literature and expert opinion. Descriptive statistics and qualitative analyses were used.
Results
An international expert panel reached consensus on 17 self-management behaviors focusing on: stable phase (n=5): pharmacotherapy, vaccination, physical activity, avoiding stimuli and smoking cessation; periods of symptom deterioration (n=1): early detection; during an exacerbation (n=5): early detection, health care contact, self-treatment, managing stress/anxiety and physical activity; during recovery (n=4): completing treatment, managing stress/anxiety, physical activity and exercise training; and after recovery (n=2): awareness for recurrent exacerbations and restart of pulmonary rehabilitation.
Conclusion
This study has provided insight into expert opinion on the most relevant and feasible self-management behaviors that can be targeted and influenced before, during and after an exacerbation to exert the highest magnitude of influence on the impact of exacerbations. Future research should focus at developing more comprehensive patient-tailored interventions supporting patients in these exacerbation-related self-management behaviors.
Keywords: COPD, self-management, exacerbation, Delphi study, self-care, Delphi technique and behavior
Introduction
Chronic obstructive pulmonary disease (COPD) is a major problem for health care worldwide and the fourth leading cause of mortality.1 A characteristic of COPD is that the natural course of symptoms and airflow limitation is slowly progressive.2 This natural course is interrupted by exacerbations characterized by a sustained worsening of patients’ respiratory symptoms, which are beyond normal day-to-day variability, acute in onset and necessitates a change in regular medication.3 These exacerbations are associated with decline in lung function4 and quality of life,5,6 increased mortality,7 and increased health care utilization.8
To address the burden on both patients and society, self-management has become increasingly important in COPD care.9,10 Self-management is defined as
an individual’s ability to detect and manage symptoms, treatment, physical and psychosocial consequences, and lifestyle changes inherent in living with a chronic condition.11
COPD self-management interventions should be
structured but personalized and often multi-component, with goals of motivating, engaging and supporting the patients to positively adapt their health behavior(s) and develop skills to better self-manage their disease.12
The pivotal objective is to change health behaviors and to equip patients with skills to actively participate in the management of their disease.13 In recent years, considerable research has shown that self-management interventions have positive effects on exacerbation recovery time, reduce hospital admissions and are associated with increased quality of life.9,14 However, it remains unclear which intervention components are most effective, and at which moment in time these should be applied to reduce exacerbation impact.
Although large heterogeneity in frequency, severity, symptoms and recovery time of exacerbations is observed both between and within COPD patients,15–18 recent evidence suggests that in general, each exacerbation follows a certain pattern of different phases: from the stable phase leading to an exacerbation, followed by a recovery phase and subsequently a high-risk period for recurrent exacerbations.15,17,19 To reduce exacerbation impact, previous self-management interventions focused mainly on self-management behaviors in a specific phase, solely aiming at early detection of exacerbations and taking prompt actions14,20 or at self-management after exacerbations.21 However, patients could potentially exert more influence on the impact of exacerbations by aggregating self-management behaviors prior to, during and after an exacerbation.
To move toward more comprehensive and effective exacerbation-related self-management interventions, it is essential to identify which self-management behaviors are most relevant to reduce exacerbation impact and in which phase these should be applied. Furthermore, it is important to determine which behaviors are feasible to influence. So far, evidence regarding the most relevant and feasible self-management behaviors to reduce exacerbation impact is inconclusive. By investigating expert opinion regarding self-management behaviors, including those for which limited evidence is available, a deeper understanding of self-management behaviors can be reached.22
The aim of this study was to reach consensus with experts on the most relevant set of self-management behaviors that has the potential to maximally reduce the impact of exacerbations and is feasible to target and influence, prior to, during and after an exacerbation. This knowledge is essential for the development of future targeted and tailored self-management interventions that can potentially further reduce exacerbation impact.
Material and methods
Study design
A Delphi study, based on components of the RAND/UCLA appropriateness method,23 was performed to reach consensus on the relevance and feasibility of self-management behaviors that maximize reduction of exacerbation impact. The RAND/UCLA appropriateness method aims to combine the best available scientific evidence with the collective judgement of experts to yield a statement regarding the appropriateness of performing a medical procedure when scientific evidence is lacking.23 This meets our objective to reach consensus on self-management behaviors by integrating both explicit and tacit unpublished knowledge and perspectives of experts in the absence of conclusive evidence.22,24 This Delphi study consists of different phases of data collection and data analysis following an iterative process.24
Methods
The iterative phases of data collection and analysis of this Delphi study are presented in Figure 1. Methods were regularly discussed within the research team and subsequently adjusted in order to enhance validity.
Figure 1.
Study design of the Delphi study.
Development of a conceptual model
At first, a scoping literature review was performed to specify different symptom fluctuation phases during the course of COPD and identify aims for each phase regarding reduction of exacerbation impact. First, articles suggested by COPD experts, and relevant studies from their reference lists, were reviewed.3,15,17,19,25,26 Subsequently, a conceptual model of patients’ fluctuations in symptoms during the course of COPD was developed by 2 researchers (YJGK and JCAT) (Figure 2). This model distinguishes 5 phases and is in line with previous studies presenting symptom fluctuations over time.3,26 Finally, overarching aims regarding reduction of exacerbation impact were formulated by our research team for each phase (Figure 2).
Figure 2.
Conceptual model of patients’ fluctuations in symptoms during the course of COPD.
Literature review
Second, a scoping literature review was performed to identify potential self-management behaviors aiming to reduce exacerbation impact in each phase of the conceptual model. Potential self-management behaviors were identified following a stepwise procedure. An initial set of self-management behaviors was generated by our research team. Relevant literature was searched to substantiate these behaviors and to identify additional behaviors. Literature was searched for strategies aiming to reduce exacerbation impact according to the following steps: 1) relevant COPD guidelines; 2) (systematic) reviews and 3) longitudinal studies (randomized controlled trials [RCTs]/cohort studies). Database searches were performed in PubMed, ScienceDirect and Google Scholar using the following search terms and derivatives: COPD, exacerbation, self-management, prevention, treatment and recovery. Studies were selected when published between 2000 and 2016. Articles were first screened by title and abstract on relevance followed by reading the full text. An article was included only when it focused on strategies associated with reduction of exacerbation impact (based on the phases of the conceptual model) and requiring self-management behaviors. Strategies were extracted from literature and used as intermediate outcomes for self-management behaviors. Based on the strategies found, required self-management behaviors, as supported by previous self-management studies, were formulated by our research team for each phase of the conceptual model.9,13,20,21,25 An overview of the selected literature can be found in Appendix 1.
Selection of experts
A purposive sample of international respiratory experts was selected for the face validity round and the Delphi study. The experts included in the face validity round were all invited to participate in the Delphi study. The aim was to include a panel of 15 experts for the Delphi study, since a panel of this size is considered sufficient to create diversity regarding representation, while being small enough to include solely key experts in this area.23,27 Based on an expected response rate of 60%,28 30 experts were invited to participate.
Diversity of perspectives was pursued by selecting a heterogeneous and multidisciplinary panel of medical doctors (pulmonologists and general practitioners with specific interest in COPD) and key-researchers in the field of COPD. Inclusion criteria were: proven expertise on COPD exacerbations and/or COPD self-management and willingness to participate in the study. Two inclusion criteria were later added: at least 5 publications on the topic (researchers) and extensive COPD patient contact (medical doctors). Experts were selected based on their publications or connections with the research team. One expert of the panel was involved in this study by providing expert opinion on the process of the Delphi study (TWE).
Eligible experts were informed about the study and invited to participate by email. According to the Dutch law, a study with experts does not require any legal assessment by an Institutional Review Board. Experts were informed that the participation was voluntary and anonymous. Informed consent was obtained at each Delphi round.
Face validity
The conceptual model, including potential relevant self- management behaviors, was initially discussed in face-to-face meetings with international respiratory experts (n=5) to determine face validity (YJGK and JCAT) to: 1) verify whether the conceptual model adequately reflected symptom fluctuation phases during the COPD course; 2) identify whether the selected behaviors were relevant; and 3) to investigate if relevant behaviors were missing. These expert meetings were structured by a topic list (Appendix 2). Based on this step, several behaviors were added to the conceptual model. No self-management behaviors were excluded. Additional literature was then consulted to substantiate the added self-management behaviors (YJGK and JCBS). The conceptual model was used to develop surveys for the Delphi rounds.
Delphi rounds
In total, 2 Delphi surveys were developed in the online survey service SurveyMonkey (SurveyMonkey Inc., San Mateo, CA, USA); (YJGK and JCBS). Experts received an invitation to participate in the study by email, including the online survey for the first round. Background information on the study objectives, a survey instruction and questions about demographic characteristics were included in the survey. Experts were asked to complete the survey within 3 weeks. After round 1, experts received an overview of the results and were asked to complete the second online survey within 3 weeks. During both rounds, a reminder was sent 2 weeks after the initial mailing.
The aim of the first survey was to assess the relevance and feasibility of the predetermined self-management behaviors for each phase of the conceptual model and gain insight into the degree of consensus between experts.23,27,29 Each behavior was rated on 3 statements: 1) the association between a behavior and reduction of exacerbation impact in a specific phase (relevance), 2) the extent to whether there is room for improvement (relevance to intervene on a behavior) and 3) the feasibility to influence a behavior. The relevance of a behavior was determined by both statement 1 and 2. Based on the RAND method, all statements were scored on a 9-point Likert scale (1= strongly disagree, 9= strongly agree).23 An example of a survey question is shown in Appendix 3. behaviors that were considered to be relevant in more than one phase were assessed only once on all 3 statements. only statement one, identifying the relevance of a behavior, was rated when a behavior returned in subsequent phases.
Open questions were asked at each phase to gain more in-depth understanding of the ratings, investigate whether relevant behaviors were missing, and check whether methodological inconsistencies were present. To anticipate missing data, participants were obliged to complete all ratings on a survey page before continuing to the next page. The survey was pilot-tested by 5 experts in the field. In addition, a linguist of English language verified the survey.
Based on the RAND-method, self-management behaviors were classified into 3 different categories based on median scores: not relevant/feasible (median 1–3), uncertain (median 4–6) and relevant/feasible (median 7–9).23 An interquartile range (IQR) was calculated to determine the level of consensus between experts, with smaller IQR values indicating higher degree of consensus.30,31 An IQR ≤2 was considered as consensus among experts,27,32 meaning that at least 50% of all ratings are situated within 2 points around the median rating of the expert panel.33 The inclusion and exclusion criteria for the final behaviors are presented in Table 1. A behavior was considered to be relevant and feasible when at least 2 statements had a median score of 7–9 (including statement 1), no statement had a median score of 1–3 and all 3 statements had an IQR ≤2. All statements without consensus (IQR >2) were presented again to the expert panel in Delphi round 2.
Table 1.
Inclusion and exclusion criteria of self-management behaviors
| Inclusion or exclusion | Criteria |
|---|---|
| Include in final list | Median score of 7–9 for at least 2 statements, including statement 1* AND an IQR ≤2 for all 3 statements AND no statement with a median score of 1–3. |
| Include in next Delphi round | Each statement with an IQR >2. |
| Exclude | • 1 (or more) statement(s) with a median of 1–3 AND an IQR ≤2 for all 3 statements. |
| • 2 (or more) statements with a median of 4–6 AND an IQR ≤2 for all 3 statements. | |
| • Statement 1 with a median of 4–6 AND an IQR ≤2 for all 3 statements. |
Notes:
A median of 7–9 on statement 1 was required in all cases since statement 1 investigated the association between a self-management behavior and reduction of exacerbation impact and was considered to be most important. Median 1–3= not relevant/feasible; median 4–6= uncertain; median 7–9= relevant/feasible.
Abbreviation: IQR, interquartile range.
The aim of the second Delphi survey was to move toward consensus on selection of relevant and feasible self-management behaviors. Furthermore, the aim was to gain more understanding into the level of disagreement between experts.23 The survey consisted of behaviors without consensus based on Delphi round 1 and new behaviors that were suggested by experts in round 1. Experts were asked to re-rate the statements without consensus. For each statement, the median score, IQR, and relevant remarks from round 1 were shown. Experts were asked to provide comments when their score deviated from the median from round 1. This survey was again pilot-tested by 3 experts in the field. Same descriptive statistics and inclusion and exclusion criteria (Table 1) were used in this round. Ratings obtained from round 2 with consensus were used as final ratings. It was decided that if consensus could not be reached after 2 Delphi rounds, a third round would be initiated.
Data analysis
Data analysis was supported by using SPSS version 22 (IBM Corporation, Armonk, NY, USA).34 Descriptive statistics were used to analyze and report all data. A qualitative analysis of comments provided by the expert panel in both online surveys was performed. First, all comments were read in full by 2 researchers (JCBS and YJGK). Second, one researcher summarized all comments from both rounds for each of the 5 phases (JCBS). This summary was subsequently reviewed to check accuracy (YJGK). Meaningful statements were verified based on consensus (JCBS and YJGK) and presented in the “Results” section.
Results
Demographic characteristics
Demographic characteristics of the Delphi panel are shown in Table 2. In total, 19 of the 30 (63%) invited experts agreed to participate in this study. One expert completed the first survey after the round 1 deadline and was therefore excluded from round 1, but did participate in round 2. The second Delphi round was completed by 16 experts. Reasons for non-response were: retirement, limited time for participation and no longer working in this field.
Table 2.
Demographic characteristics of the Delphi panel (n=19)*
| Characteristics | n (%) |
|---|---|
| Gender | |
| Male | 15 (79) |
| Age (years) | |
| 30–49 | 9 (47) |
| 50–69 | 9 (47) |
| >70 | 1 (5) |
| Experience in respiratory care focusing on COPD (years) | |
| 0–5 | 1 (5) |
| 6–15 | 7 (37) |
| 16–25 | 3 (16) |
| 26–35 | 5 (26) |
| >35 | 3 (16) |
| Area of focus | |
| Patient care | 1 (5) |
| Research | 5 (26) |
| Both patient care and research | 13 (68) |
| Country | |
| the Netherlands | 8 (42) |
| UK | 2 (11) |
| Belgium | 1 (5) |
| Canada | 1 (5) |
| USA | 4 (21) |
| Australia | 3 (16) |
| Discipline | |
| Pulmonology | 11 (58) |
| Primary care | 3 (16) |
| Nursing | 1 (5) |
| Physiotherapy | 1 (5) |
| Epidemiology | 3 (16) |
Note:
One expert of the panel was involved in this study by providing expert opinion on the process of the Delphi study.
Delphi round 1
The results of round 1 are show in Table 3. In total, 27 self-management behaviors, distributed among the 5 phases of the conceptual model, were rated in the first Delphi round.
Table 3.
Results of Delphi round 1 and 2
| Phase | Self-management behaviors rated in Delphi rounds | Behavior identified from (source) | Round 1 | Round 2 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Relevancea | Improvementb | Feasibilityc | Relevancea | Improvementb | Feasibilityc | |||||||||
| M | IQR | M | IQR | M | IQR | M | IQR | M | IQR | M | IQR | |||
| 1. Stable (low risk) | Adherence to pharmacotherapy (LABA/LAMA and/or ICS)* | Literature | 8 | 3 | 8 | 1 | 7 | 2 | 8 | 0.75 | NA | NA | NA | NA |
| Influenza vaccination uptake* | Literature | 7 | 1.25 | 6 | 2.25 | 7 | 3 | NA | NA | 6 | 2 | 7 | 0.75 | |
| Daily physical activity* | Literature | 7 | 2.25 | 9 | 1 | 7 | 1.25 | 7 | 1.5 | NA | NA | NA | NA | |
| Avoiding viral or bacterial stimuli* | Literature | 7 | 2 | 7 | 2.25 | 6.5 | 2 | NA | NA | 7 | 0.75 | NA | NA | |
| Managing exposure to air quality** | Face validity | 7 | 2 | 6 | 1.5 | 5 | 1.25 | NA | NA | NA | NA | NA | NA | |
| Smoking cessation* | Literature | 8 | 2 | 8 | 2.25 | 7 | 3 | NA | NA | 8 | 0 | 7 | 1 | |
| Managing exposure to indoor air quality** | Delphi round 1 | NA | NA | NA | NA | NA | NA | 6 | 1.75 | 5 | 1.75 | 5.5 | 1 | |
| 2. Mild deterioration | Early detection of symptom deterioration* | Face validity | 8 | 1.25 | 8 | 0.5 | 7.5 | 1 | NA | NA | NA | NA | NA | NA |
| Correct increase of SABA** | Literature | 6 | 2 | 5.5 | 2.25 | 6 | 2.25 | NA | NA | 6 | 1 | 6 | 0.75 | |
| Performing breathing exercises** | Literature | 5.5 | 2.25 | 7 | 3 | 6.5 | 2 | 5 | 1 | 7 | 0.75 | NA | NA | |
| Performing energy conservation techniques** | Literature | 6 | 1.25 | 6 | 2 | 6 | 2.25 | NA | NA | NA | NA | 6 | 0.75 | |
| 3. Exacerbation | Early detection of an exacerbation* | Literature | 9 | 1 | 8 | 2 | 7 | 1 | NA | NA | NA | NA | NA | NA |
| Prompt treatment with corticosteroids and/or AB (self-treatment)* | Literature | 8.5 | 1 | 8 | 1.25 | 7 | 1.25 | NA | NA | NA | NA | NA | NA | |
| Prompt treatment with corticosteroids and/or AB (contacting HCP)* | Literature | 8.5 | 1 | 8 | 1.5 | 8 | 2.25 | NA | NA | NA | NA | 8 | 0 | |
| Manage stress and anxiety* | Literature | 7 | 1 | 7 | 0.5 | 6 | 2 | NA | NA | NA | NA | NA | NA | |
| Correct increase of SABA** | Literature | 6 | 3.5 | NA | NA | NA | NA | 6 | 1 | NA | NA | NA | NA | |
| Performing breathing exercises** | Literature | 6.5 | 2 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | |
| Daily physical activity* | Literature | 7 | 3 | NA | NA | NA | NA | 7 | 1.75 | NA | NA | NA | NA | |
| Performing energy conservation techniques** | Literature | 6 | 2.25 | NA | NA | NA | NA | 6 | 1 | NA | NA | NA | NA | |
| 4. Recovery | Completing treatment with corticosteroids and/or AB* | Face validity | 7 | 1.25 | 6.5 | 2.25 | 7 | 1.25 | NA | NA | 6.5 | 1 | NA | NA |
| Manage stress and anxiety (concerning current event)* | Literature | 7 | 1.5 | 7 | 1.25 | 7 | 2.25 | NA | NA | NA | NA | 7 | 0.75 | |
| Adjusted exercise- and resistance training* | Literature | 7 | 1.25 | 8 | 2 | 7 | 1.25 | NA | NA | NA | NA | NA | NA | |
| Daily physical activity* | Literature | 8 | 1.5 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | |
| Correct use of SABA** | Literature | 6.5 | 3 | NA | NA | NA | NA | 6 | 0.75 | NA | NA | NA | NA | |
| Performing energy conservation techniques** | Literature | 6 | 2 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | |
| Performing breathing exercises** | Literature | 6 | 1.25 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | |
| 5. Stable (at risk) | Increased awareness for recurring exacerbation* | Literature | 8 | 2 | 8 | 2 | 7.5 | 1.25 | NA | NA | NA | NA | NA | NA |
| Early (re)start of pulmonary rehabilitation* | Face validity | 8 | 2 | 8 | 2.25 | 7 | 2 | NA | NA | 8 | 0 | NA | NA | |
Notes:
Self-management behavior included in final list after round 1 or 2 based on median scores and consensus within the expert panel.
Self-management behavior excluded after round 1 or 2 based on median scores and consensus within the expert panel.
Relevance – The association of this behavior and reducing exacerbation impact (statement 1).
Improvement – The extent to whether there is room for improvement in this behavior (statement 2).
Feasibility – The feasibility to influence this behavior (statement 3). Scores are on a 9 point Likert scale (1= strongly disagree, 9= strongly agree); bold type indicates the returning self-management behaviors in which only statement 1 was rated and therefore scores on statements 2 and 3 are not applicable.
Abbreviations: NA, not applicable; M, median; IQR, interquartile range; AB, antibiotics; HCP, health care provider; LABA, long-acting β2-agonist; LAMA, long-acting muscarinic antagonist; ICS, inhaled corticosteroids; SABA, short-acting β2-agonist.
Seven self-management behaviors were considered relevant and feasible with consensus (IQR ≤2) and were directly included in the final list. Highest ratings were observed regarding early detection of symptom deterioration/exacerbations and prompt treatment. Four behaviors were excluded after round 1. All statements without consensus (IQR >2) were included in round 2. Managing exposure to indoor air quality was added to the second round based on suggestions of 2 experts in round 1.
Delphi round 2
Table 3 shows the results of round 2 in which statements regarding 16 self-management behaviors were re-rated and the added self-management behavior was rated for the first time. Based on round 2, 10 of the remaining behaviors were included in the final list. Seven behaviors were excluded after round 2.
Final list of self-management behaviors
Based on 2 Delphi rounds, the expert panel reached consensus on a set of 17 self-management behaviors that were both relevant and feasible to reduce exacerbation impact (Table 4). The expert panel agreed on 5 behaviors in the stable phase (low risk), 1 during mild deterioration of symptoms, 5 during an exacerbation, 4 during exacerbation recovery and 2 in the stable phase (at risk). Daily physical activity was considered to be relevant in 3 phases (1, 3 and 4). Based on relations between behaviors, the final behaviors were subdivided into 9 categories (Table 4).
Table 4.
Final self-management behaviors per phase of the conceptual model
| Phase | Statement | Relevant and feasible self-management behaviors | Categories | ||
|---|---|---|---|---|---|
| 1 | 2 | 3 | |||
| 1. Stable phase (low risk) | ✓ | ✓ | Adherence to pharmacotherapy (LABA/LAMA and/or ICS) | Adherence to pharmacotherapy | |
| ✓ | ~ | ✓ | Influenza vaccination uptake | Influenza vaccination | |
| ✓ | ✓ | ✓ | Daily physical activity | Physical activity/exercise | |
| ✓ | ✓ | ~ | Avoiding viral or bacterial stimuli | Avoiding stimuli | |
| ✓ | ✓ | ✓ | Smoking cessation | Smoking cessation | |
| 2. Mild deterioration | ✓ | ✓ | ✓ | Early detection of symptom deterioration | Early detection of symptom deterioration |
| 3. Exacerbation (including onset) | ✓ | ✓ | ✓ | Early detection of an exacerbation | Early detection of symptom deterioration |
| ✓ | ✓ | ✓ | Prompt treatment corticosteroids and/or AB (self-treatment) | Medical treatment of exacerbations | |
| ✓ | ✓ | ✓ | Prompt treatment corticosteroids and/or AB (contact with HCP) | Medical treatment of exacerbations | |
| ✓ | ✓ | ~ | Manage stress and anxiety | Manage stress and anxiety | |
| ✓ | NA | NA | Daily physical activity | Physical activity/exercise | |
| 4. Recovery phase | ✓ | ~ | ✓ | Completing treatment of antibiotics and/or corticosteroids | Medical treatment of exacerbations |
| ✓ | ✓ | ✓ | Manage stress and anxiety (concerning current event) | Manage stress and anxiety | |
| ✓ | ✓ | ✓ | Adjusted exercise- and resistance-training | Physical activity/exercise | |
| ✓ | NA | NA | Daily physical activity | Physical activity/exercise | |
| 5. Stable phase (at risk) | ✓ | ✓ | ✓ | Increased awareness for recurring exacerbation | Awareness for recurrent exacerbations |
| ✓ | ✓ | ✓ | Early (re)start of pulmonary rehabilitation | Physical activity/exercise | |
Notes: Bold type indicates the returning self-management behaviors in which only statement 1 was rated. ✓ = Median of 7–9 for statement. ~ = Median of 4–6 for statement. 1= Relevance – The association of this behavior and reducing exacerbation impact (statement 1). 2= Improvement – The extent to whether there is room for improvement in this behavior (statement 2). 3= Feasibility – The feasibility to influence this behavior (statement 3).
Abbreviations: AB, antibiotics; HCP, health care provider; LABA, long-acting β2-agonist; LAMA, long-acting muscarinic antagonist; ICS, inhaled corticosteroids; NA, not applicable.
Qualitative results
By asking open questions, experts commented on their ratings. No methodological inconsistencies regarding the survey were identified based on experts’ remarks. Meaningful comments on self-management behaviors are described and further explained by citations in Table 5.
Table 5.
Qualitative results of Delphi round 1 and 2
| Phase | Results | Self-management behavior | Citations |
|---|---|---|---|
| 1. Stable (low risk) | Experts initially disagreed on the relevance of pharmacotherapy (LABA/LAMA and/or ICS). One expert stated that high level evidence is available, whereas other experts stated that ICS use is overrated in many COPD patients and that the role of pharmacotherapy is small and only relevant to a subgroup of patients. Furthermore, experts initially disagreed on the room for improvement regarding influenza vaccination uptake. Some experts considered vaccination uptake to be high, whereas others noted that room for improvement depends on region and country. Disagreement on the feasibility to influence vaccination uptake was observed since some experts believed that influenza vaccination is already heavily promoted and uptake is hard to influence. [C1] Regarding managing exposure to air quality, experts agreed on the relevance to reduce exacerbation impact, but explained doubts regarding the feasibility to influence this behavior. [C2] Finally, for smoking cessation, comments mainly focused on the feasibility to influence this behavior. Experts believe that this is feasible and that we should keep trying; however, they also stated that current smokers often show resistance regarding quitting smoking and that it is hard to influence this behavior. [C3] |
Vaccination uptake | I think that vaccination uptake is harder to influence than other pharmacotherapy. People may have stronger opinions why to do/not to do it. [C1] |
| Managing exposure to air quality | I have my doubts. Patients do need to get out sometimes, and if there are prolonged periods of air pollution, I am not sure how to solve this problem. [C2] | ||
| Quitting/cutting down smoking | Smokers with COPD are a population that often have difficulty quitting and therefore will likely require more intensive interventions to influence smoking rates. [C3] | ||
| 2. Mild deterioration | Several comments focused on the relevance of correctly increasing SABA in several phases. Some experts considered SABA as obsolete and one expert believed it should not be stimulated at all. Moreover, the role of SABA was questioned regarding further symptom deterioration. [C4] Regarding breathing exercises, some experts believed that these are only relevant for very few patients and were not impressed by the strength of evidence. [C5] As well as by SABA, experts believed that performing breathing exercises leads to symptom relief, but does not influence further symptom deterioration. Moreover, experts questioned whether patients are aware of these techniques. Finally, energy conservation techniques were not considered to be relevant since experts believed that this is a more general self-management strategy and not specifically aimed at preventing exacerbations. In addition, experts believed that most patients automatically perform this behavior. [C6] |
Correct increase of SABA | […] whereas the use of SABA will lead to symptom relief, it will not (directly) influence further deterioration of the exacerbation. [C4] |
| Performing breathing exercises | Little is known about the individual need for breathing exercises. It is certainly not for all patients. [C5] | ||
| Performing energy conservation techniques | Most patients will conserve energy to reduce their symptoms during early exacerbations as a natural behavior. [C6] | ||
| 3. Exacerbation | Experts directly agreed on the relevance of early detection of symptom deterioration and exacerbations in round 1. Experts believed that symptom diaries or symptom scoring with action plans can be supportive. Nevertheless, the complexity of this behavior was emphasized as well as the importance of personalized advice regarding this behavior by focusing on previously experienced symptoms with exacerbations. [C6 and 7] Initially there was disagreement between experts on the relevance of daily physical activity in both the stable phase (low risk) and exacerbation phase. Some experts stated that there is proven evidence for stimulating daily physical activity and that this is important to reduce exacerbation impact. [C8] Others were uncertain about the evidence or even believed that daily physical activity might worsen a patient’s situation when experiencing an exacerbation. [C9] Furthermore, one expert pointed out that it is uncertain when physical activity should be initiated. [C10] Finally, although there was consensus on the feasibility to influence self-treatment at exacerbation onset, some experts have commented on this. Experts were confident that self-treatment is feasible although not for all COPD patients, considering health literacy, motivation, cognitive levels and relatively high age in this population. [C11] |
Early detection of an exacerbation | Collaborative self-management aimed at the early detection of the exacerbation and prompt, appropriate treatment of that exacerbation is reasonable, but this is not easily achieved. [C6] I think generic non-personalized advice will fail. These are recurrent exacerbations and emphasis should be on: ‘When you feel you are beginning to get bad, like last time […]. [C7] |
| Daily physical activity | In my opinion, daily PA is important to prevent the vicious circle of breathlessness and to increase fitness and health (and thereby preventing exacerbations). [C8] The exacerbation will cause reduced activity; and encouraging the patient, who is so restricted, to increase their activity might be counter-productive and perhaps even worsen the situation (especially given common comorbidities like cardiac disease). [C9] Although physical activity is likely important, there continues to be uncertainty of when to initiate physical activity and how intensive it should be. [C10] |
||
| Self-treatment with corticosteroids and/or AB | I think that self-treatment is indicated for a select group of patients. Otherwise overtreatment is a potential risk. [C11] | ||
| 4. Recovery | Relaxation techniques were considered to be important to manage stress and anxiety at exacerbation onset. However, initially disagreement was observed in the feasibility to change this behavior during exacerbation recovery. One expert thought that stress may be challenging to overcome, while another expert believed that well-designed interventions can influence stress and anxiety. [C12] | Manage stress and anxiety | With good quality interventions/support stress and anxiety can be influenced; at least partly. [C12] |
| 5. Stable (at risk) | Early (re)start of pulmonary rehabilitation was considered to be relevant and feasible, although initially consensus on room for improvement was lacking and challenges towards feasibility to influence this behavior were reported. One expert specifically stated that there is substantial room for improvement in this behavior based on current patient performance, but this might be out of the scope of the patients own influence or health care provider’s influence. [C13] | Early (re)start of pulmonary rehabilitation | No this is health care providing politics. [C13] |
Abbreviations: LABA, long-acting β2-agonist; LAMA, long-acting muscarinic antagonist; ICS, inhaled corticosteroids; SABA, short-acting β2-agonist; C, citation from expert; PA, physical activity; AB, antibiotics.
Discussion
To our knowledge, this is the first study providing insight into expert opinion on the most relevant and feasible self-management behaviors that can be targeted and influenced, prior to, during and after an exacerbation, to reduce the impact of COPD exacerbations. Based on 2 Delphi rounds, consensus within the expert panel was reached on a set of 17 self-management behaviors that were perceived both relevant and feasible to target and influence. According to our study results, self-management should focus on adherence to pharmacotherapy, influenza vaccination, physical activity/exercise, avoiding stimuli, smoking cessation, early detection of symptom deterioration, medical treatment of exacerbations, managing stress and anxiety, and awareness for recurrent exacerbations. This study shows that each symptom fluctuation phase requires aggregating specific self-management behaviors from COPD patients to maximally reduce the impact of exacerbations.
Self-management behaviors included in our final list were mostly in line with other studies. The importance of early detection of (recurrent) exacerbations and taking prompt actions was emphasized by the high ratings of our expert panel showing that there is still large room for improvement and that these behaviors are feasible to influence.14,20,21 Furthermore, the importance of adherence to pharmacotherapy, influenza vaccination, smoking cessation, physical activity and exercise training is in line with recommendations in international guidelines.2,35,36
In contrast, correct increase of short-acting β2-agonist (SABA) use, performing breathing techniques and energy conservation techniques were expected to be relevant2,35,37,38 but were not included in our final list. This may be explained by our specific study focus on reducing exacerbation impact. Whereas, according to the experts, increasing SABA and performing breathing techniques were not expected to contribute to the reduction of exacerbation impact, experts considered this important for symptom relief. In addition, breathing exercises may not be relevant for all patients (as stated by one of our experts). Evidence regarding breathing techniques is also contradictory as one review showed that pursed lip breathing is effective to improve dyspnea,38 while 2 other reviews found no evidence that breathing exercises improve lung function or relieve symptoms.37,39
Furthermore, managing exposure to air quality was rated in our Delphi study since recent evidence shows that air pollution is related to an increased risk for exacerbations.40 Additionally, managing exposure to “indoor” air quality was added based on expert suggestions. Although there was consensus between experts on the relevance of both behaviors, they were not included in the final list since room for improvement and feasibility to influence these behaviors were considered to be insufficient, which may be explained by the limited available evidence.
Surprisingly, avoiding viral or bacterial stimuli was considered to be important by our expert panel while convincing evidence is still lacking. The latter may be explained by the limited evidence regarding exacerbations cause,2 and the fact that this behavior is difficult to measure and challenging to influence. Furthermore, managing stress and anxiety was considered to be important during an exacerbation and recovery, while no convincing evidence was found regarding reduction of exacerbation impact. This might be explained by the fact that anxiety affects symptom perception and self-management13,41 and by the feasibility to influence this behavior.42 Finally, notable was that daily physical activity was considered to be both relevant and feasible to influence before, during and after an exacerbation. This is in line with evidence showing that maintaining daily activity prevents exacerbations and that lower physical activity levels are associated with an increased risk of exacerbation-related hospitalizations.2,43,44 However, it is important to take into account that some experts in our panel believed that daily physical activity may worsen a patient’s situation during an exacerbation and that it is uncertain when physical activity should be initiated.
Strengths and limitations
An important strength of this study was the tailored and iterative study design. Extensive literature review on self-management behaviors and a face validity check on the predetermined behaviors contributed to the validity of the study, as well as pilot-testing of the surveys. Furthermore, our specific focus on identifying the most promising self-management behaviors was strengthened by rating both relevance and feasibility of behaviors according to 3 statements. Moreover, validity was enhanced by including a small expert panel of key international experts in the field focusing on both research and patient care.
However, this study has also some limitations. Little foundation for our methodological decisions can be provided, given the great variation in the methodological designs of Delphi studies. In addition, our study did not include a face-to-face meeting with experts between the Delphi rounds to discuss ratings, investigate areas of disagreement, and gain more in-depth insights. We did, however, include open questions at each phase to gain more understanding into the ratings and to check whether methodological inconsistencies were present. Furthermore, our study focused on predetermined self-management behaviors derived from literature that might have caused experts to be less open-minded to introduce new behaviors themselves. To prevent missing additional input, a comment box was available to introduce new behaviors. Finally, it is important to note that experts’ ratings on statement 2 and 3 of the Delphi survey may be influenced by local factors as, for example, the local health care context or policy in countries individual experts are living in. Therefore, supporting the identified self-management behaviors might require different strategies dependent on the local setting.
Implications for practice and future research
The findings of this study are important for both professionals providing self-management support and researchers focusing on the development of self-management interventions. This study highlights the importance of focusing on a tailored set of self-management behaviors in each phase of the course of COPD, so that patients can exert the highest magnitude of influence on the impact of exacerbations. This study therefore strengthens the need for multi-component interventions.
Future research should focus on the development of more comprehensive and tailored self-management interventions based on available evidence. As self-management requires behavior change in COPD patients, it is important to identify effective intervention components aiming at behavior change. Recently, increased attention has been paid on determining intervention strategies that are effective in targeting the relevant mechanisms of change using behavior change techniques. Therefore, it is important to investigate patients’ capabilities, opportunity and motivation to perform specific behaviors identified by this study.45
In future development of more comprehensive interventions eHealth or mHealth opportunities should be explored, since both provide possibilities to strongly individualize and tailor self-management interventions during the whole course of COPD.45
Conclusion
This study identified a set of 17 relevant and feasible self-management behaviors that can be targeted and influenced to maximize reduction of exacerbation impact based on consensus within an international expert panel. To exert the highest magnitude of influence on the impact of exacerbations, it is important that patients perform specific self-management behaviors before, during and after an exacerbation, that focus on adherence to pharmacotherapy, influenza vaccination, physical activity/exercise, avoiding stimuli, smoking cessation, early detection of symptom deterioration, medical treatment of exacerbations, managing stress and anxiety, and awareness for recurrent exacerbations. Future research should focus on developing and evaluating more comprehensive interventions supporting patients in these specific exacerbation-related self-management behaviors. Proven effective interventions addressing these behaviors should be considered in COPD care.
Acknowledgments
We thank all of the experts who participated in this Delphi study for their time and sharing their expertise.
Footnotes
Disclosure
The authors report no conflict of interest in this work.
References
- 1.Lopez AD, Shibuya K, Rao C, et al. Chronic obstructive pulmonary disease: current burden and future projections. Eur Respir J. 2006;27(2):397–412. doi: 10.1183/09031936.06.00025805. [DOI] [PubMed] [Google Scholar]
- 2.Global Initiative for Chronic Obstructive Lung Disease (GOLD) Global Strategy for the Diagnosis, Management, and Prevention of Chronic Obstructive Pulmonary Disease [updated 2016] [Accessed February 2, 2016]. Available from: http://goldcopd.org/global-strategy-diagnosis-management-prevention-copd-2016/
- 3.Rodriguez-Roisin R. Toward a consensus definition for COPD exacerbations. Chest. 2000;117(5 Suppl 2):398S–401S. doi: 10.1378/chest.117.5_suppl_2.398s. [DOI] [PubMed] [Google Scholar]
- 4.Donaldson GC, Seemungal TA, Bhowmik A, Wedzicha JA. Relationship between exacerbation frequency and lung function decline in chronic obstructive pulmonary disease. Thorax. 2002;57(10):847–852. doi: 10.1136/thorax.57.10.847. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Llor C, Molina J, Naberan K, Cots JM, Ros F, Miravitlles M. Exacerbations worsen the quality of life of chronic obstructive pulmonary disease patients in primary healthcare. Int J Clin Pract. 2008;62(4):585–592. doi: 10.1111/j.1742-1241.2008.01707.x. [DOI] [PubMed] [Google Scholar]
- 6.Miravitlles M, Ferrer M, Pont A, et al. IMPAC Study Group Effect of exacerbations on quality of life in patients with chronic obstructive pulmonary disease: a 2 year follow up study. Thorax. 2004;59(5):387–395. doi: 10.1136/thx.2003.008730. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Soler-Cataluña JJ, Martínez-García MÁ, Sánchez PR, Salcedo E, Navarro M, Ochando R. Severe acute exacerbations and mortality in patients with chronic obstructive pulmonary disease. Thorax. 2005;60(11):925–931. doi: 10.1136/thx.2005.040527. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Vestbo J, Hurd SS, Agustí AG, et al. Global strategy for the diagnosis, management, and prevention of chronic obstructive pulmonary disease GOLD executive summary. Am J Respir Crit Care Med. 2013;187(4):347–365. doi: 10.1164/rccm.201204-0596PP. [DOI] [PubMed] [Google Scholar]
- 9.Zwerink M, Brusse-Keizer M, van der Valk PDLPM, et al. Self management for patients with chronic obstructive pulmonary disease. Cochrane Database Syst Rev. 2014;3(3):CD002990. doi: 10.1002/14651858.CD002990.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Bourbeau J, van der Palen J. Promoting effective self-management programmes to improve COPD. Eur Respir J. 2009;33(3):461–463. doi: 10.1183/09031936.00001309. [DOI] [PubMed] [Google Scholar]
- 11.Barlow J, Wright C, Sheasby J, Turner A, Hainsworth J. Self-management approaches for people with chronic conditions: a review. Patient Educ Couns. 2002;48(2):177–187. doi: 10.1016/s0738-3991(02)00032-0. [DOI] [PubMed] [Google Scholar]
- 12.Effing TW, Vercoulen JH, Bourbeau J, et al. Definition of a COPD self-management intervention: international Expert Group consensus. Eur Respir J. 2016;48(1):46–54. doi: 10.1183/13993003.00025-2016. [DOI] [PubMed] [Google Scholar]
- 13.Bourbeau J, Nault D. Self-management strategies in chronic obstructive pulmonary disease. Clin Chest Med. 2007;28(3):617–628. doi: 10.1016/j.ccm.2007.06.002. [DOI] [PubMed] [Google Scholar]
- 14.Trappenburg JCA, Monninkhof EM, Bourbeau J, et al. Effect of an action plan with ongoing support by a case manager on exacerbation-related outcome in patients with COPD: a multicentre randomised controlled trial. Thorax. 2011;66(11):977–984. doi: 10.1136/thoraxjnl-2011-200071. [DOI] [PubMed] [Google Scholar]
- 15.Aaron SD, Donaldson GC, Whitmore GA, Hurst JR, Ramsay T, Wedzicha JA. Time course and pattern of COPD exacerbation onset. Thorax. 2012;67(3):238–243. doi: 10.1136/thoraxjnl-2011-200768. [DOI] [PubMed] [Google Scholar]
- 16.Donaldson GC, Seemungal TA, Patel IS, Lloyd-Owen SJ, Wilkinson TM, Wedzicha JA. Longitudinal changes in the nature, severity and frequency of COPD exacerbations. Eur Respir J. 2003;22(6):931–936. doi: 10.1183/09031936.03.00038303. [DOI] [PubMed] [Google Scholar]
- 17.Hurst JR, Vestbo J, Anzueto A, et al. Evaluation of COPD Longitudinally to Identify Predictive Surrogate Endpoints (ECLIPSE) Investigators Susceptibility to exacerbation in chronic obstructive pulmonary disease. N Engl J Med. 2010;363(12):1128–1138. doi: 10.1056/NEJMoa0909883. [DOI] [PubMed] [Google Scholar]
- 18.Jenkins CR, Celli B, Anderson JA, et al. Seasonality and determinants of moderate and severe COPD exacerbations in the TORCH study. Eur Respir J. 2012;39(1):38–45. doi: 10.1183/09031936.00194610. [DOI] [PubMed] [Google Scholar]
- 19.Hurst JR, Donaldson GC, Quint JK, Goldring JJP, Baghai-Ravary R, Wedzicha JA. Temporal clustering of exacerbations in chronic obstructive pulmonary disease. Am J Respir Crit Care Med. 2009;179(5):369–374. doi: 10.1164/rccm.200807-1067OC. [DOI] [PubMed] [Google Scholar]
- 20.Walters J, Turnock A, Walters E, Wood-Baker R. Action plans with limited patient education only for exacerbations of chronic obstructive pulmonary disease. Cochrane Database Syst Rev. 2010;(5):CD005074. doi: 10.1002/14651858.CD005074.pub3. [DOI] [PubMed] [Google Scholar]
- 21.Harrison SL, Janaudis-Ferreira T, Brooks D, Desveaux L, Goldstein RS. Self-management following an acute exacerbation of COPD: a systematic review. Chest. 2015;147(3):646–661. doi: 10.1378/chest.14-1658. [DOI] [PubMed] [Google Scholar]
- 22.Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–1015. [PubMed] [Google Scholar]
- 23.Fitch K, Bernstein SJJ, Aguilar MDD, et al. The RAND/UCLA Appropriateness Method: User’s Manual. 2001. [Accessed February 23, 2016]. Available from: https://www.rand.org/content/dam/rand/pubs/monograph_reports/2011/MR1269.pdf.
- 24.McKenna HP. The Delphi technique: a worthwhile research approach for nursing? J Adv Nurs. 1994;19(6):1221–1225. doi: 10.1111/j.1365-2648.1994.tb01207.x. [DOI] [PubMed] [Google Scholar]
- 25.Wilkinson TM, Donaldson GC, Hurst JR, Seemungal TA, Wedzicha JA. Early therapy improves outcomes of exacerbations of chronic obstructive pulmonary disease. Am J Respir Crit Care Med. 2004;169(12):1298–1303. doi: 10.1164/rccm.200310-1443OC. [DOI] [PubMed] [Google Scholar]
- 26.Seemungal TAR, Donaldson GC, Bhowmik A, Jeffries DJ, Wedzicha JA. Time course and recovery of exacerbations in patients with chronic obstructive pulmonary disease. Am J Respir Crit care Med. 2000;161(5):1608–1613. doi: 10.1164/ajrccm.161.5.9908022. [DOI] [PubMed] [Google Scholar]
- 27.Birko S, Dove ES, Ozdemir V, Dalal K. Evaluation of nine consensus indices in delphi foresight research and their dependency on delphi survey characteristics: a simulation study and debate on delphi design and interpretation. PLoS One. 2015;10(8):1–14. doi: 10.1371/journal.pone.0135162. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Asch DA, Jedrziewski MK, Christakis NA. Response rates to mail surveys published in medical journals. J Clin Epidemiol. 1997;50(10):1129–1136. doi: 10.1016/s0895-4356(97)00126-1. [DOI] [PubMed] [Google Scholar]
- 29.Rayens MK, Hahn EJ. Building consensus using the delphi policy. Policy Polit Nurs Pract. 2000;1(4):308–315. [Google Scholar]
- 30.University of Leicester Measures of variability: the range, interquartile range and standard deviation. 2009. [Accessed March 18, 2016]. pp. 1–7. Available from: http://www2.le.ac.uk/offices/ld/resources/numerical-data/variability.
- 31.Manikandan S. Measures of dispersion. J Pharmacol Pharmacother. 2011;2(4):315. doi: 10.4103/0976-500X.85931. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Linstone H, Turoff M. The Delphi Method: Technique and Applications. Reading MA: Addison-Wesley; 1975. [Google Scholar]
- 33.Von der Gracht HA. Consensus measurement in Delphi studies. Review and implications for future quality assurance. Technol Forecast Soc Change. 2012;79(8):1525–1536. [Google Scholar]
- 34.IBM Corp . Released 2013. IBM SPSS Statistics for Windows, Version 22.0. Armonk, NY: IBM Corp; [Google Scholar]
- 35.Criner GJ, Bourbeau J, Diekemper RL, et al. Prevention of acute exac-erbations of COPD. Chest. 2015;147(4):894–942. doi: 10.1378/chest.14-1676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Spruit MA, Singh SJ, Garvey C, et al. ATS/ERS Task Force on Pulmonary Rehabilitation An official American thoracic society/European respiratory society statement: Key concepts and advances in pulmonary rehabilitation. Am J Respir Crit Care Med. 2013;188(8):e13–e64. doi: 10.1164/rccm.201309-1634ST. [DOI] [PubMed] [Google Scholar]
- 37.Hill K, Patman S, Brooks D. Effect of airway clearance techniques in patients experiencing an acute exacerbation of chronic obstructive pulmonary disease: a systematic review. Chron Respir Dis. 2010;7(1):9–17. doi: 10.1177/1479972309348659. [DOI] [PubMed] [Google Scholar]
- 38.Facchiano L, Snyder CH, Núñez DE. A literature review on breathing retraining as a self-management strategy operationalized through Rosswurm and Larrabee’s evidence-based practice model. J Am Acad Nurse Pract. 2011;23(8):421–426. doi: 10.1111/j.1745-7599.2011.00623.x. [DOI] [PubMed] [Google Scholar]
- 39.Aaron SD. Management and prevention of exacerbations of COPD. BMJ. 2014;349(2):g5237. doi: 10.1136/bmj.g5237. [DOI] [PubMed] [Google Scholar]
- 40.Hansel NN, McCormack MC, Kim V. The effects of air pollution and temperature on COPD. COPD. 2016;13(3):372–379. doi: 10.3109/15412555.2015.1089846. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Disler RT, Gallagher RD, Davidson PM. Factors influencing self-management in chronic obstructive pulmonary disease: an integrative review. Int J Nurs Stud. 2012;49(2):230–242. doi: 10.1016/j.ijnurstu.2011.11.005. [DOI] [PubMed] [Google Scholar]
- 42.Tselebis A, Pachi A, Ilias I, et al. Strategies to improve anxiety and depression in patients with COPD: a mental health perspective. Neuropsychiatr Dis Treat. 2016;12:297–328. doi: 10.2147/NDT.S79354. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Spruit MA, Pitta F, McAuley E, ZuWallack RL, Nici L. Pulmonary rehabilitation and physical activity in patients with chronic obstructive pulmonary disease. Am J Respir Crit Care Med. 2015;192(8):924–933. doi: 10.1164/rccm.201505-0929CI. [DOI] [PubMed] [Google Scholar]
- 44.Moy ML, Teylan M, Weston NA, Gagnon DR, Garshick E. Daily step count predicts acute exacerbations in a US cohort with COPD. PLoS One. 2013;8(4):19–23. doi: 10.1371/journal.pone.0060400. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.West R, Michie S. A Guide to Development and Evaluation of Digital Behaviour Change Interventions in Healthcare. London: Silverback Publishing; 2016. [Google Scholar]