Mixed Methods Research in the Development and Evaluation of Complex Interventions in Palliative and End-of-Life Care: Report on the MORECare Consensus Exercise

Abstract

Background: Complex interventions are common in palliative and end-of-life care. Mixed methods approaches sit well within the multiphase model of complex intervention development and evaluation. Generic mixed methods guidance is useful but additional challenges in the research design and operationalization within palliative and end-of-life care may have an impact on the use of mixed methods.

Objective: The objective of the study was to develop guidance on the best methods for combining quantitative and qualitative methods for health and social care intervention development and evaluation in palliative and end-of-life care.

Methods: A one-day workshop was held where experts participated in facilitated groups using Transparent Expert Consultation to generate items for potential recommendations. Agreement and consensus were then sought on nine draft recommendations (DRs) in a follow-up exercise.

Results: There was at least moderate agreement with most of the DRs, although consensus was low. Strongest agreement was with DR1 (usefulness of mixed methods to palliative and end-of-life care) and DR5 (importance of attention to respondent burden), and least agreement was with DR2 (use of theoretical perspectives) and DR6 (therapeutic effects of research interviews). Narrative comments enabled recommendation refinement. Two fully endorsed, five partially endorsed, and two refined DRs emerged. The relationship of these nine to six key challenges of palliative and end-of-life care research was analyzed.

Conclusions: There is a need for further discussion of these recommendations and their contribution to methodology. The recommendations should be considered when designing and operationalizing mixed methods studies of complex interventions in palliative care, and because they may have wider relevance, should be considered for other applications.

Introduction

Qualitative and quantitative methodologies answer different research questions. Combining and integrating quantitative and qualitative research methods from the different research paradigms of positivism and interpretivism in a mixed method approach can provide a more comprehensive view,¹ potentially generating greater knowledge yield: “a whole greater than the sum of the parts.”² Complex interventions, such as health service and social care interventions,³ are common in palliative and end-of-life care and often require multiple research questions to develop and evaluate them. The use of mixed methods is therefore an appropriate means to answer such questions and a mixed method approach sits well within the multiphase model of complex intervention development and evaluation⁴ advocated by the Medical Research Council (MRC) framework.^3,5,6

Numerous generic mixed methods resources exist, and there are an increasing number of examples of mixed methods studies conducted in palliative and end-of-life care (for examples of both, see Farquhar et al 2011⁴). However, insufficient experience with this approach in palliative and end-of-life care research exists to support any best practice recommendations that are sufficiently detailed to be practically applied. The infancy of mixed methods in this field was well demonstrated by a review by Flemming et al, which identified just one published randomized controlled trial (RCT) that included a qualitative element among 146 palliative care RCTs.⁷ Guidance will develop as more palliative and end-of-life care research with an integrated mixed methods approach is designed, delivered, and available for evaluation.

Generic guidelines are very useful, however there are challenges in the design and operationalization of studies within palliative and end-of-life care that affect the use of mixed methods. To address this, a one-day workshop, organized as part of the MRC and National Institute for Health Research (NIHR) Project MORECare,⁸ brought together a diverse group of experts in mixed methods and palliative and end-of-life care research. The workshop and follow-up consensus exercise were intended to create guidance on the best methods for combining and integrating quantitative and qualitative methods for health service and social care intervention development and evaluation in palliative and end-of-life care.

Methods

Participants

Potential workshop delegates from across the United Kingdom were identified from the published literature on mixed methods and palliative and end-of-life care research. We sought approximately 30 delegates and expected a 50% response rate. All workshop delegates and the MRC MORECare Programme Advisory Group (PAG) were also invited to participate in a follow-up consensus exercise (short web-based survey).

Procedure

The study was approved by the University of Manchester Ethics Committee. Informed consent was obtained from all participants. The workshop began with a brief context-setting introduction defining complex interventions,³ outlining the MRC framework for the development and evaluation of complex interventions,^3,5,6 giving definitions of palliative⁹ and end-of-life care¹⁰ (see Table 1), and discussing six key challenges (recruitment, attrition, differing disease trajectories, respondent burden, randomization, and outcomes) of palliative and end-of-life care research identified from the published literature^11–45 that may impact the use of mixed methods (see Table 2).

Table 1.

Working Definitions of Palliative and End-of-Life Care

Concepts	Working definitions used in workshop
Palliative care	“An approach which improves the quality of life of patients and their families facing life-threatening illness, through the prevention, assessment and treatment of pain and other physical, psychosocial and spiritual problems”9
End-of-life care	Usually the care of a person during the last part of their life, from the point at which it has become clear that the person is in a progressive state of decline
	The care of a person lasting longer than the period during which someone is considered to be dying10

Table 2.

Six Key Challenges of Palliative and End-of-Life Care Research that May Impact the Use of Mixed Methods

Challenges	Outline of potential difficulties
Recruitment	Participant identification can be difficult due to varying definitions of palliative and end-of-life care11 and their application to varying health service systems12 and patient registers. In addition, heterogeneity of the palliative and end-of-life care population means that not all patients “fit” study criteria.12,13
	Studies can face difficulties from patients' inability to participate due to deteriorating health status12,14–15 or death prior to recruitment.12 There can also be an inability to provide informed consent.12,14–17
	Gatekeeping can occur when well-intentioned clinicians12,14,17,18–20 or family members15 seek to protect patients from research participation at the end of life, despite research suggesting that some even very ill patients21–22 and bereaved relatives23 find it helpful and would like the opportunity to participate. This gatekeeping can impact on the efficiency of research, as well as its validity and reliability due to potentially biased samples.18,20
	Gatekeeping can also occur by ethics committees.18,24
Attrition	Attrition (including drop-out, nonresponse, withdrawal, and protocol deviation) is a major issue in studies with a follow-up element.11–13,25–28 Some attrition is inevitable in end-of-life care research due to deterioration in a patient's condition or death, but might also result from respondent burden and changes, such as in respondents' circumstances (e.g., relocation) or contextual changes (e.g., changes in local services or treatments). This can lead to missing data that can affect validity. Unavoidable attrition due to patient deterioration or death can result in the use of proxies29–32 and retrospective data,33 with their own associated issues of validity and reliability.
Differing disease trajectories	The palliative care population is not a homogenous group.12 Working with different disease trajectories (broadly classified into cancer, organ failure, and frail elderly or dementia trajectories) presents challenges for intervention34 and research design. Heterogeneity also exists within the broad disease trajectories, e.g., variations by cancer type.
Respondent burden	Working with very ill patients with high symptom burden and busy informal caregivers makes respondent burden an important consideration12,35–36); the length and complexity of data collection instruments and methods require careful thought. Similar to attrition, respondent burden can result in the use of proxies and retrospective data, with their associated issues of validity and reliability.29–33,36
Randomization	Randomization to a control group may deny patients or caregivers access to an intervention.14,35,37–38 This can sometimes be overcome by the use of cluster designs19,37 or fast-track/waiting list trial designs,37–38 but the ability to use these designs depends on where patients are on those varying disease trajectories.
Outcomes	Because palliative and end-of-life care research investigate an approach that seeks to improve the quality of life of patients, outcomes may be subjective and multidimensional.12–13,26,36,39–42 Outcomes often need to relate to families.12 Difficulties in the administration and interpretation of outcome measures have been reported.43–45 These issues present particular challenges for designing studies and powering trials and respondent burden.

Three presentations were given by experienced mixed methods researchers on (i) mixed methods approaches to evaluating health care; (ii) combining qualitative and quantitative methods to give a fuller answer to research questions; and (iii) including qualitative methods in trials of health and social care interventions. Each was followed by open discussion with delegates.

Delegates then participated in one of three facilitated groups of 10–12 people. In order to ensure coverage of all types of studies, each group focused on a different phase(s) of the MRC framework: Group 1 on Pre-Clinical and Phase I studies, Group 2 on Phase II and Phase III randomized controlled trials, and Group 3 on Phase IV Implementation Studies. Group allocation was based on a combination of delegates' prior preferences depending on their area of expertise and a balance of group size. Within their groups, delegates were asked to identify potential recommendations for guidance on integrating qualitative and quantitative methods in the development and evaluation of complex interventions in palliative and end-of-life care in relation to their phase(s) of the MRC framework.

Each group was moderated by an experienced facilitator who explained the task and then kept delegates on task but stimulated discussion of issues as they arose. Facilitators asked participants to each spend ten minutes writing potential recommendations on cards, based on their expertise and experience. The cards were then read by each delegate to the group. Potential recommendations were entered on a laptop for screen projection to facilitate discussion. Each group then clarified and rephrased items as required. Any cards not considered to be duplicates that had not been read out were saved for later transcription. Each group was audiotaped, with permission, using digital recorders. The recordings enabled later synthesis of items into potential recommendations by the research team. Thus the Transparent Expert Consultation process was used,⁴⁶ based on a modified nominal group technique. Nominal group techniques have a long history of use in health and medicine, providing a structured format for discussion and synthesis of information, with the aim of rapidly producing solutions or decisions.⁴⁷

Following the workshop, items for potential recommendations from across the three groups were combined: over 200 items had been generated. The numerous duplicate items were removed by members of the research team, followed by items that were too generic, rather than specific to or pertaining particularly to palliative and end of life care research. The remaining items were synthesised into potential recommendations. A reduced list of nine draft recommendations was thus generated for presentation to all workshop participants and all members of the MRC MORECare PAG via a brief online consultation. Participants scored their level of agreement for each draft recommendation on a scale 1–9 (1, low; 9, high) using an anonymous web-based survey; written comments were also solicited.

Analysis

Descriptive statistics (frequencies and medians) and plots (box plots of inter-quartile ranges) were used to summarise and analyse the ratings of agreement, and grouped by level of agreement and level of consensus (see Table 3). Narrative comments were read in order to provide meaning to the quantitative ratings given; they were then summarized for reporting and informed the refinement of the nine draft recommendations. The recommendations were then considered in relation to the six identified key challenges of palliative and end-of-life care research upon which mixed methods may impact.

Table 3.

Classification of Agreement and Consensus on Recommendations

Rating	Median	Range
Strong agreement/high consensus	≥8	<2
Strong agreement/low consensus	≥8	≥2
Moderate agreement/high consensus	<8–>6	<2
Moderate agreement/low consensus	<8–>6	≥2
No agreement/high consensus	≥4–≤6	<2
No agreement/low consensus	≥4–≤6	≥2
Moderate disagreement/high consensus	<4–>2	<2
Moderate disagreement/low consensus	<4–>2	≥2
Strong disagreement/high consensus	≤2	<2
Strong disagreement/low consensus	≤2	≥2

Results

Thirty-three delegates took part in the one-day workshop. These included 17 predominantly palliative and end-of-life care researchers and 16 health services researchers out of 67 invited: of the 34 nonparticipants, 9 had initially agreed but then had to withdraw. For the online follow-up consultation, 58 people were eligible and a 45% response rate was achieved (n=26/58), following a reminder at two weeks.

Table 4 lists the nine DRs and the median levels of agreement achieved by each in the online consultation. Figure 1 shows the range of levels of agreement and interquartile ranges for each draft recommendation.

Table 4.

Nine Draft Recommendations Included in the Online Consensus, Median Level of Agreement, Summary of Narrative Comments, and Resulting Refined Recommendations

Draft recommendation (DR), arranged in order of agreement	Median level of agreement* (IQR) and classification of agreement	Summary of narrative comments for each draft recommendation	Resulting fully endorsed recommendation (R), partially endorsed recommendation (PR), or refined draft recommendation (RDR) (Changes from draft recommendations in italics)
DR1: Mixed methods (integrating quantitative and qualitative methods) research is a particularly useful approach for palliative and end of life care research.	8 (7–9) Strong agreement/low consensus	In the complex dynamic context of palliative care, mixed methods can offer a means to develop feasible, acceptable and effective interventions and enhance the evidence base required; this is true of other fields too. Patient and family views are particularly informative, but either method used separately cannot give a full picture. However, the choice of method always depends on the research question and each method needs to be justified. The greatest challenge lies in integrating analyses. Some respondents also commented that this item was not phrased as a recommendation.	R1: Mixed methods (integrating quantitative and qualitative methods) research is a particularly useful approach for palliative and end of life care research: the exact choice of method will depend on the research question and each method needs to be justified.
DR5: The degree of respondent burden needs careful consideration in palliative and end-of-life care research, including prioritization of key outcome measures, qualitative questions, and splitting data collection.	8 (7–9) Strong agreement/low consensus	Place of data generation and mode should also be considered. Piloting is important and users are a good source of advice. However, we should not assume patients are too tired: they often find participation fulfilling and meaningful thus decisions about burden should be taken with participants not for them.	R5: The degree of respondent burden needs careful consideration in palliative and end of life care research and researchers should consider prioritization of key outcome measures and qualitative questions, whether splitting data collection sessions may be necessary, and the place and mode of data collection. Piloting and user involvement inform respondent burden concerns but decisions about burden should be taken with participants and not for them.
DR4: Wherever possible qualitative exploration of experiences of participation in randomized controlled trials and other “well designed studies” should be carried out.	7 (6–9) Moderate agreement/low consensus	Asking participants about experiences of RCT participation and ways of improving research design is useful, particularly where in relation to RCTs and where RCTs are rare (as in palliative and end-of-life care), however there may be a cost implication and the additional potential respondent burden should be justified: it is not needed for every trial. “Participants” might include clinicians as well as patients and carers. Qualitative exploration of experiences of the intervention is more important but interviewer effects may be higher in palliative and end of life care than in other fields (see R06) thus skilful design and execution of qualitative components is required.	PR4: Where justified, qualitative exploration of experiences of participation in randomized controlled trials and other “well-designed studies” should be carried out and should include all participant groups, e.g., patients, carers, and clinicians; this is in addition to qualitative exploration of experiences of the intervention.
DR7: Trial registers need to include fields for registration of qualitative components of the study or parallel qualitative studies.	7 (6–9) Moderate agreement/low consensus	May be particularly useful where new qualitative methods are being established, and may even be useful for non-trials. However including such additional fields would not necessarily mean that a qualitative study has been undertaken.	PR7: Trial registers need to include fields for registration of qualitative components of the study or parallel qualitative studies and similar nontrial registers should be established.
DR3: Palliative and end-of-life care research requires a multidisciplinary approach and thus teams will need to be comprised of requisite disciplines (clinical and academic), to answer the research question proposed.	7 (5.75–9) Moderate agreement/low consensus	Rather than requiring it, palliative and end-of-life care research benefits from a multi-disciplinary approach. Multi-disciplinarity develops realistic research with academic and clinical significance however some palliative and end of life care research is non-clinical thus team composition depends on the research question. Equality and respect of skills and knowledge within multi-disciplinary teams is required.	PR3: Depending on the research question, palliative and end-of-life care research may benefit from a multidisciplinary team-working approach, and thus in most cases teams will need to consist of requisite disciplines (clinical and academic), to answer the research question proposed.
DR9: Greater emphasis is required on implementation studies in palliative and end of life care research.	7 (5.75-9) Moderate agreement/low consensus	Implementation is important and may be a natural development of the specialty as it establishes. Some respondents queried the wording of this item: greater than what?	PR9: Greater emphasis is required on implementation studies in palliative and end-of-life care research than is presently the case. This may develop naturally as the specialty establishes, but researchers should be encouraged to move from Phase III RCTs to Phase IV implementation studies and to consider the contribution a mixed methods approach could make to them.
DR8: Researchers working on mixed methods studies need both quantitative and qualitative skills (or training), but given the sensitivity of research in palliative and end of life care research, researchers need additional empathy and communication.	6.5 (4.75-8.25) Moderate agreement/low consensus	Involvement of separate quantitative and qualitative research professionals is important for research design. Further, it may be more appropriate to employ different researchers working together to provide the different skills required, but with an openness to, and understanding of, each paradigm. Teaching and identification of such skills, and those of sensitivity, is challenging. These needs may not be unique to palliative and end of life care research, however support and debriefing of all members of the research team may be important in this and similar fields.	PR8: Researchers working on mixed methods studies need both quantitative and qualitative skills (or training) which should come from separate relevant professionals at the design stage and potentially also at the data collection stage, but given the sensitivity of research in palliative and end of life care research, researchers conducting interviews need additional empathy and communication skills. Researchers from differing paradigms need an openness to, and understanding of, other paradigms within the team. Support and debriefing of all team members is important.
DR2: Given the current state of (under) development of palliative and end-of-life care research, the explicit use of theoretical perspective should be employed from the outset.	6 (5-8) No agreement/low consensus	There should be theoretical considerations in any research. The explicit use of theory should be encouraged rather than insisted upon. Theory is relevant to both quantitative and qualitative research thus knowledge of both areas is required. If paucity of palliative and end of life care research exists investigators should be open to developing new theoretical frameworks for the field taking a grounded theory approach; use theory at the outset when research is deductive or data at the outset when research is inductive. Some respondents queried the meaning of this item.	RDR2: Given the current state of (under) development of palliative and end-of-life care research, the explicit use of theoretical perspective is encouraged from the outset and investigators should be open to developing new theoretical frameworks for the field.
DR6: Given the nature of sensitivities involved in palliative and end-of-life care research there are potentially particular problems of therapeutic effects of research interviews and the Hawthorne effect.	6 (4.75-8) No agreement/low consensus	The Hawthorne effect is common to all settings and data collection methods. Therapeutic effects are possible confounding factors but there is a lack of evidence of their nature and duration. This may, in itself, need to be the subject of research, particularly in the context of palliative and end of life care. Some respondents commented that this item was not phrased as a recommendation and that it included two components.	RDR6: Given the nature of sensitivities involved in palliative and end of life care research there are potentially particular problems of therapeutic effects of research interviews which can be confounding: this should be considered when designing studies and interpreting findings. The lack of evidence of the nature and duration of therapeutic effects requires further research.

Number of respondents=26;

^*Potential range 1–9: 1=strongly disagree; 9=strongly agree

DR=draft recommendation; IQR=interquartile range; R=fully endorsed recommendation; PR=partially endorsed recommendation; RDR=refined draft recommendation; RCT=randomized controlled trial

FIG. 1.

Box plot of the Interquartile ranges and medians of levels of agreement for the nine recommendations (box:25th and 75th percentiles).

When considered in relation to Table 3, Table 4, and Figure 1, the strongest agreement (but low consensus) was with DR1 (the usefulness of mixed methods to palliative and end-of-life care research) and DR5 (the importance of attention to respondent burden), and the lowest level of agreement and consensus was with DR2 (the use of theoretical perspective) and DR6 (therapeutic effects of research interviews). Thus only DR1 and DR5 could be transitioned into fully endorsed recommendations (Rs; R1 and R5), five DRs could be transitioned into partially endorsed recommendations (PRs; PR3, PR4, PR7, PR8, and PR9), and two remained as DRs (DR2 and DR6).

Table 4 also provides a summary of the narrative comments in relation to each DR. Respondent comments linked to lower ratings for a DR sought clarification of recommendations (e.g., in relation to DR2 [the explicit use of theoretical perspective]), stated that methodology choice depended on the research question (e.g., in relation to DR3 [multidisciplinary team] and DR4 [exploration of the participation experience]), or expressed the view that the area was not problematic either in general or in the specialist field of palliative and end-of-life care research (e.g., DR5 [respondent burden], DR6 [therapeutic effects of research interviews], and DR8 [researcher skill-sets]). In light of the strength of agreement and the narrative comments, the final column of Table 4 presents refined versions of the two Rs, five PRs, and two DRs (then becoming refined draft recommendations [RDRs]). Figure 2 summarizes the entire process from initial item generation to the final transition into fully endorsed, partially endorsed, and refined draft recommendations.

FIG. 2.

Transition from items for potential recommendations to full recommendations.

A further set of narrative comments generated at the end of the online consultation raised important points, such as the need to consider mixed methods in palliative and end-of-life care studies other than those relating to the development and evaluation of complex interventions; the generic nature of many of the recommendations; the importance of the timing of participant recruitment in relation to the challenge of illness trajectories in palliative and end-of-life care; and the likely variation in the importance of recommendations depending on the audience, e.g., some will be more important for commissioners of research.

Due to the range of expertise among workshop delegates, the workshop had commenced with a context-setting presentation of six key challenges of palliative and end-of-life care research. The results of an analysis of the relationship of these key challenges to each recommendation (Rs, PRs, and RDRs) are provided in Table 5.

Table 5.

Relationship of the Recommendations to the Six Challenges of Palliative and End-of-Life Care Research

Fully endorsed recommendation (R), partially endorsed recommendation (PR), and refined draft recommendation (RDR)	Relationship of recommendations to six key challenges of palliative and end-of-life care research
R1Mixed methods (integrating quantitative and qualitative methods) research is a particularly useful approach for palliative and end-of-life care research: the exact choice of method will depend on the research question and each method needs to be justified.	R1 has relevance for the challenges of recruitment, attrition, respondent burden, and outcomes. For recruitment and attrition, mixed methods could provide valuable data to inform the design of recruitment and sample-retention strategies and may be useful in addressing issues of gatekeeping (qualitative methods could explore how and why this occurs and quantitative methods could assess the impact of strategies to reduce it). R1 relates to the challenges of respondent burden and outcomes by stating explicitly the need for methods to apply to the research question, ensuring relevance. The holistic, sometimes individualized, nature of palliative care interventions may heighten the need to explore why an intervention did or did not work and the potential for targeting interventions. Mixed methods may be particularly suited to addressing such questions.
R5The degree of respondent burden needs careful consideration in palliative and end-of-life care research, and researchers should consider prioritization of key outcome measures and qualitative questions, whether splitting data collection sessions may be necessary, and the place and mode of data collection. Piloting and user involvement inform respondent burden concerns, but decisions about respondent burden should be taken with participants and not for them.	R5 has relevance for the challenges of attrition, differing disease trajectories, respondent burden, and outcomes. Attrition may result from respondent burden and may be compounded by the impact of differing disease trajectories. The appropriate choice and prioritization of outcome measures is important in all studies but perhaps enhanced in mixed methods studies where multiple outcomes may occur. The practice of PR3 (multidisciplinary teams) and PR4 (identifying the experiences of participation) could further inform R5.
PR4Where justified, qualitative exploration of experiences of participation in randomised controlled trials and other “well designed studies” should be carried out and should include all participant groups e.g. patients, carers, and clinicians; this is in addition to qualitative exploration of experiences of the intervention.	PR4 has relevance for all six of the challenges: recruitment, attrition, differing disease trajectories, respondent burden, randomization, and outcomes. Given the relative infancy of palliative and end-of-life care research, there is a role for qualitative and quantitative methods in exploring what participation in palliative and end-of-life care research is like and how it can be improved. This relates to all phases of the MRC framework, but there is a particular opportunity within feasibility trials that can address quantitative questions relating to response, unblinding and missing data rates as well as qualitative questions relating to the participation experience. The findings of such mixed method explorations could inform strategies for the six challenges.
PR7Trial registers need to include fields for registration of qualitative components of the study or parallel qualitative studies, and similar nontrial registers should be established.	PR7 has the ability to enable the dissemination of information and facilitate reflection, thus it has relevance for all six of the challenges: recruitment, attrition, differing disease trajectories, respondent burden, randomization, and outcomes. Andrew and Halcomb48 have called for more discourse in health disciplines in order to advance methodological underpinnings of mixed methods health research, examine the newly emerging typologies of health study designs, investigate the integration of health data sets, and present practical advice regarding conduct and dissemination of mixed methods health research. Including mixed method designs in study registers would contribute to this endeavor.
PR3Depending on the research question, palliative and end-of-life care research may benefit from a multidisciplinary team-working approach; thus in most cases teams will need to consist of requisite disciplines (clinical and academic) to answer the research question proposed.	PR3 has relevance for the challenges of recruitment, attrition, differing disease trajectories, respondent burden, and outcomes. The expertise of multidisciplinary teams could usefully inform strategies to address recruitment and attrition. Clinical expertise can inform the impact of differing disease trajectories on intervention and study design and ways to address these, and both clinical and academic experts can work together to consider respondent burden and the identification of relevant outcomes in relation to research questions.
PR9Greater emphasis is required on implementation studies in palliative and end-of-life care research than is presently the case: this may develop naturally as the specialty establishes, but researchers should be encouraged to move from Phase III RCTs to Phase IV implementation studies and consider the contribution a mixed methods approach could make to them.	PR9 has relevance for the challenges differing disease trajectories and outcomes. Palliative and end-of-life care interventions that are developed and evaluated within one broad disease group may or may not be generalizable to other broad disease groups; even those developed within one broad disease group (e.g., cancer) but focused on particular disease sites (e.g., lung) may or may not be relevant for other sites of that disease (e.g., colorectal cancer). Thus taking studies beyond Phase III RCTs into Phase IV implementation studies would inform on the generic resonance of interventions in terms of disease groups, as well as the success of interventions in different contexts. Given the type of research questions that are posed in implementation studies (e.g., relating to service change and adoption of interventions), the study designs (e.g., case studies) and outcomes required may be more qualitative in nature but may still benefit from a mixed methods approach in order to achieve comprehensiveness.
PR8Researchers working on mixed methods studies need both quantitative and qualitative skills (and training) that should come from separate relevant professionals at the design stage and potentially also at the data collection stage, but given the sensitivity of research in palliative and end-of-life care, researchers conducting interviews need additional empathy and communication skills. Researchers from differing paradigms need an openness to, and understanding of, other paradigms within the team. Support and debriefing of all team members is important.	PR8 has relevance for the challenges of attrition, differing disease trajectories, and respondent burden. Having the relevant skills in mixed methods study design and management as well as in quantitative and qualitative data collection (either as a multidisciplinary interviewing team or as individual interviewers) could have an impact on avoidable attrition and respondent burden. In terms of data collection, an awareness of differing disease trajectories and the likely impact of these on both patients and caregivers, as well as the issue of respondent burden, may be facilitated in those with skills of sensitivity, empathy, and communication. The practice of PR3 (multidisciplinary teams) and PR4 (identifying the experiences of participation) could further inform PR8.
RDR2Given the current state of (under-)development of palliative and end-of-life care research, the explicit use of theoretical perspective is encouraged from the outset, and investigators should be open to developing new theoretical frameworks for the field.	RDR2 has relevance for the challenge of outcomes. Theory is relevant to both quantitative and qualitative research and can influence our choice of outcome measures and contents of qualitative topic guides. Theory can help steer or focus data collection when we have to make use of limited data or data collection opportunities. Thus it may also relate to respondent burden.
RDR6Given the nature of the sensitivities involved in palliative and end–of-life care research, there are potentially particular problems of therapeutic effects of research interviews that can be confounding: this should be considered when designing studies and interpreting findings. The lack of evidence of the nature and duration of therapeutic effects requires further research.	RDR6 has relevance to the challenge of outcomes. Studies using mixed methods will collect data using quantitative and qualitative methods, but the dearth of methodological studies on the impact of these methods on participants, as well as the potential for qualitative methods in particular to have a confounding therapeutic effect (which may, for example, dilute the identifiable effect of an intervention across trial arms), means that robust evidence to inform mixed methods study design in this field is lacking. In the absence of this evidence, the timing of the collection of different data types requires consideration. Mixed methods studies might include methodological elements to build the evidence-base for this potential issue.

Discussion

There was moderate agreement among the respondents with the majority of the draft recommendations. This relatively modest level of agreement may relate to the subjective nature of qualitative and mixed methods research, or to the fact that not all of the initial delegation participated in the consensus exercise, which suggests that further development of the recommendations may be warranted. The highest level of agreement was with the usefulness of mixed methods to palliative and end-of-life care research (DR1) and attention to respondent burden (DR5), and the lowest level of agreement was with the use of theoretical perspective (DR2) and the therapeutic effects of research interviews (DR6). Thus only DR1 and DR5 were transitioned into fully endorsed recommendations (R1 and R5), five DRs were transitioned into partially endorsed recommendations (PR3, PR4, PR7, PR8, and PR9), and two remained as DRs requiring further debate but which we were able to refine (RDR2 and RDR6). Our analysis of the recommendations in relation to the six identified key challenges of palliative and end-of-life care research suggests that mixed methods should not be used in an attempt to address these challenges; rather it suggests that mixed methods can be appropriately used in relation to relevant research questions, but that the impact of mixed methods on these six key challenges should be considered, and that the recommendations may facilitate this.

Variations on five of these nine recommendations (i.e., R1, RDR2, PR3, R5, and PR8) may be found in generic texts on mixed methods. What this study adds, however, is that the wording of these five recommendations and the accompanying commentary presented here highlight those features of mixed methods research that should be considered when designing mixed methods studies within the particular context of palliative and end-of-life care: how to apply them and why. Thus these five recommendations are adaptations to existing mixed methods guidance. However, the remaining four (PR4: exploration of the participant experience; RDR6: therapeutic effects of research interviews; PR7: study registers to include mixed methods; and, PR9: contribution of mixed methods to implementation studies) are less often discussed in the established mixed methods literature and could thus be considered as potential new additions to even generic mixed methods guidance.

Reflection on the workshop process and outcome

The workshop generated a vast number of potential recommendations. Although most but not all were relevant to mixed methods research, the majority were not related to palliative and end-of-life care research and the challenges they face. This lack of focus on palliative and end-of-life care research may have reflected the lack of experience of some delegates in this specialized field, despite the context-setting presentation at the start of the workshop. In addition, a few of the potential recommendations generated did not take the form of recommendations but were individual words written on the cards. This varying level of development of the potential recommendations within the groups probably relates to the limited time for the group-work rather than any failings of the task set or its facilitation.

One of the limitations of this study is that the recommendations developed and their subsequent ratings may only reflect the views of those who participated in the study. All of the recommendations, regardless of their level of endorsement, will therefore benefit from wider discussion and application for their further refinement and uptake, in the spirit of calls for greater discourse.^4,47

Conclusion

The results of the workshop and consensus exercise presented here suggest further discussion and consideration of these recommendations when designing and operationalizing mixed methods studies of complex interventions in palliative and end-of-life care research. These recommendations may not be regarded as entirely exclusive to palliative or end-of-life care research, but recommendations that are particularly helpful for palliative or end-of-life care research because of their known challenges. They may therefore inform other areas of research that face similar challenges and seek to apply mixed methods in addressing appropriate research questions. The recommendations may also be regarded as relating to studies other than those focused on the development and evaluation of complex interventions and so may have wider relevance.

This study identifies a need for robust evidence on the nature and duration of therapeutic (and potentially confounding) effects of interviewing and the need to consider that recommendations may vary in their importance depending on the research question, its context, and its audience.

Acknowledgments

MORECare is funded by the NIHR and managed by the MRC as part of the Methodology Research Programme (grant reference number G0802654/1). MORECare aims to identify, appraise, and synthesize best practices to develop and evaluate palliative and end-of-life care, particularly focusing on complex service-delivery interventions and reconfigurations.

We acknowledge Alicia O'Cathain, Elizabeth Murray, and Simon Lewin for presenting at the workshop; Catherine Walshe for additional facilitating; Ellie Anscombe for additional scribing; Mark Pilling for analysis of quantitative data; Stacey Body and Susie Edwards for administrative support; all delegates for participation in the workshop, and all respondents to the online consultation exercise; and the Cicely Saunders Institute for hosting the workshop on behalf of MORECare.

The principal investigator of this study is Irene J. Higginson, and the co-principal investigator is Chris Todd. The members of MORECare are co-investigators Peter Fayers, Gunn Grande, Richard Harding, Matthew Hotopf, Penney Lewis, Paul McCrone, Scott Murray, and Myfanwy Morgan; the project advisory group is Massimo Costantini, Steve Dewar, John Ellershaw, Claire Henry, William Hollingworth, Philip Hurst, Tessa Inge, Jane Maher, Irene McGill, Elizabeth Murray, Ann Netten, Sheila Payne, Roland Petchey, Wendy Prentice, Deborah Tanner, and Celia A. Taylor; the researchers are Hamid Benalia, Catherine J. Evans, Marjolein Gysels, Nancy J. Preston, and Vicky Short.

The workshop was funded by MORECare, CECO, and COMPASS. Morag Farquhar was funded by Macmillan Cancer Support.

Author Disclosure Statement

No conflict of interest has been declared by the authors.