A review of patient and carer participation and the use of qualitative research in the development of core outcome sets

Abstract

Background

To be meaningful, a core outcome set (COS) should be relevant to all stakeholders including patients and carers. This review aimed to explore the methods by which patients and carers have been included as participants in COS development exercises and, in particular, the use and reporting of qualitative methods.

Methods

In August 2015, a search of the Core Outcomes Measures in Effectiveness Trials (COMET) database was undertaken to identify papers involving patients and carers in COS development. Data were extracted to identify the data collection methods used in COS development, the number of health professionals, patients and carers participating in these, and the reported details of qualitative research undertaken.

Results

Fifty-nine papers reporting patient and carer participation were included in the review, ten of which reported using qualitative methods. Although patients and carers participated in outcome elicitation for inclusion in COS processes, health professionals tended to dominate the prioritisation exercises. Of the ten qualitative papers, only three were reported as a clear pre-designed part of a COS process. Qualitative data were collected using interviews, focus groups or a combination of these. None of the qualitative papers reported an underpinning methodological framework and details regarding data saturation, reflexivity and resource use associated with data collection were often poorly reported. Five papers reported difficulty in achieving a diverse sample of participants and two reported that a large and varied range of outcomes were often identified by participants making subsequent rating and ranking difficult.

Conclusions

Consideration of the best way to include patients and carers throughout the COS development process is needed. Additionally, further work is required to assess the potential role of qualitative methods in COS, to explore the knowledge produced by different qualitative data collection methods, and to evaluate the time and resources required to incorporate qualitative methods into COS development.

Introduction

Clinical trials in health care provide important evidence of the efficacy and safety of interventions and treatments [1], thereby informing future patient care, clinical guidelines and health policy [2, 3]. The selection of outcomes to be measured and reported is an important part of the trial design process. Historically, the selection of outcomes has usually been based on the views of individual study teams informed by clinical and statistical considerations, and guided by regulatory considerations [4–6]. This is problematic since outcomes that matter to key stakeholders, including patients and carers, may be omitted. Furthermore, a wide variety of outcomes may be measured and reported across trials in the same health area “making it difficult or impossible to synthesise the results of different studies” [6] (p1). The difference in outcomes used across studies can also make it hard to detect reporting bias, where authors fail to report all findings because of the desire to report only positive results [7]. Greater uniformity in the reporting of outcomes and measures within a research area would help to facilitate the comparison and synthesis of research findings [7, 8].

One potential solution to this problem is the use of core outcome sets (COS). A COS is an agreed standardised set of outcomes to be measured and reported as a minimum in all trials in a specific health related area [6]. If implemented, a COS may help to ensure that outcomes are relevant to a range of stakeholders and will provide consistent trial outcome data to inform evidence synthesis, clinical practice, shared-decision making and health policy [9, 10]. Stakeholders can include patients, carers, patient representatives and patient advocates (reported as patients and carers from this point onwards), as well as healthcare professionals and decision makers including: funders, researchers, statisticians, health economists and pharmaceutical company representatives (reported as health professionals from this point onwards) [4, 10]. Ultimately, it is the patients and those around them (carers) who benefit from improved healthcare and so it is important that their views and preferences are heard, particularly as there is evidence to suggest that patients’ perspectives may differ from those of clinicians [11].

Despite the potential benefits of including a wide range of stakeholders in COS development, evidence to-date demonstrates that relatively few (18%) COS exercises include patients and carers and the reporting of the process does not always make it clear how they have participated [6]. Gargon et al. [6] recommended that further work is carried out to identify effective methods of eliciting patient and carer views in outcome set development. Data generated using qualitative methods can help to provide in-depth insight into patients’ and carers’ perspectives [12]. Therefore, qualitative methods could be well placed to identify outcomes that are important to patients and carers and to understand why that is. The aim of this review was therefore to: 1) review the methods by which patients and carers have been included as participants in COS development and 2) explore and describe the reported use of qualitative research with patients and carers. For the purpose of this review we were interested in participation, that is where patients and carers contribute data to COS development exercises as research participants, rather than involvement, where they contribute to the research process as an active research partner or advisor.

Methods

Data source

The Core Outcome Measures in Effectiveness Trials (COMET) Initiative “aims to bring together people interested in the development, reporting and promotion of COS, derived using rigorous consensus methods” [4]. The COMET Initiative database is an international repository of studies relevant to the development of COS, planned, ongoing and completed [13]. The COMET database was developed based on a systematic review using extensive searches [4] and the COMET Initiative conduct an annual search of the literature in order to keep the database up to date [13]. In addition, planned and ongoing COS exercises can be submitted to COMET by individuals or groups for inclusion in the database [13].

Search strategy

Given that the COMET database is a comprehensive source for COS development studies with the contents regularly updated, we limited our search to this one database.

COS Studies involving patient/carer involvement

The COMET database was searched on the 13 August 2015 using the following search categories: Carer organisations / Support group representatives, Charities, Conference participants, Consumers (caregivers), Consumers (patients), Families, Individuals with a known interest, Patient / Support group representatives, Service users, Guideline developers, with a study type of COS. Inclusion criteria were: papers developing COS with patients and carers as research participants. In addition, reference searches of the included papers were conducted.

Qualitative studies to inform COS development involving patients and carers

From these searches we identified all papers that described research using qualitative data collection methods (e.g. focus groups, interviews).

Data extraction and reporting

A data extraction pro-forma was developed, piloted and used to record study specific information: title, author, year, location of study (country), health area and data collection dates. For COS papers not using qualitative data collection methods we extracted the data collection methods used and the number of health professionals and patient and carer representatives participating in these. For papers reporting qualitative data collection methods we extracted; the stated qualitative methodological framework and rationale for this (please note: in qualitative research the methodological framework guiding research conduct, such as grounded theory, phenomenology or ethnography is distinguished from the methods used during conduct e.g. sampling, data collection, analytic approach) [14]; methods (sampling approach; data collection and analysis); sample characteristics; resource use (costs, resources and time involved); stated strengths and limitations and stated impact of the qualitative research). Some data items were informed by the CASP Qualitative Checklist [15].

The lead author (JJ) extracted data from all included papers and a second researcher (JMM, LLJ, MJC or TJHK) checked the extraction for accuracy on all papers reporting the use of qualitative methods. Any discrepancies were resolved via discussion within the research team. Data extraction were combined when a COS development exercise was reported across more than one paper. Data have been summarised descriptively. Recommendations have been made for the transparent reporting of qualitative research to inform COS development.

Results

COS studies with patients and carers

Included papers

Of the 666 records on the COMET database, our initial search strategy returned 149 papers (Fig 1). Seventy-three of the returned studies were unpublished, and of the remaining 76, 24 were excluded because there were insufficient details to determine whether they were reporting a COS including patients or carers. Through reference searches of the included papers an additional seven papers [16–22] were identified as part of the included COS development exercises. Of these, four [16, 19, 21, 22] were not archived on COMET at the point of the final search and three [17, 18, 20] papers were not returned in the initial search because they did not include patients or carers as participants; however, they described part of a COS pathway which included patient or carers reported in a separate publication.

Fig 1. Flow diagram of paper identification and inclusion process.

Participants in the COS development

The 59 included papers represent a total of 34 COS development exercises covering 32 different health areas (Table 1). In 19 papers the number of participants was unclear.

Table 1. Summary of included papers.

Reference	Publication year	Health area	Ethics committee approval obtained	Data collection methods	Health Professionals (n)	Patients (n)	Carers and/or representatives (n)
Core outcome set development exercises using qualitative methods with patients, carers and representatives
Allard et al [16]α	2014	Neurodisability	Not stated	Focus groups	0	50	47
				Interviews	0	4	6
Janssens [17] et alα	2014	Neurodisability	Not stated	Delphi	233	0	0
Morris et al [23] et al	2015	Neurodisability	Not stated	OtherΩ	7	3	5
Arnold et al [24]	2008	Fibromyalgia	Yes	Focus groups	0	48	0
Mease et al [25]	2008	Fibromyalgia	Yes	Delphi	23	100	0
Mease et al [26]	2007	Fibromyalgia	Yes	Consensus meeting	Not stated	Not stated	0
Mease et al [27]	2009	Fibromyalgia	Not stated	Consensus conference	Not stated	Not stated	0
Stamm et al [21]α	2009	Osteoarthritis	Yes	Focus groups	0	56	0
Kloppenburg et al [28]	2014	Osteoarthritis	Not stated	Delphi	31	0	0
				OtherΩ	Not stated	Not stated	0
Potter et al [19]απ	2013	Breast cancer	Yes	Interviews	35	31	0
Potter et a [20]α	2014	Breast cancer	Yes	Interviews	35	0	0
Potter et al [29]	2015	Breast cancer	Yes	Delphi	88	215	0
				Consensus meeting	23	15	0
Sanderson et al [30]	2010a	Rheumatoid arthritis	Yes	Interviews	0	23	0
Sanderson et al [31]	2010b	Rheumatoid arthritis	Yes	Nominal group technique	0	26	0
				Survey	0	254	0
Swigris et al [22]α	2005	Interstitial lung disease (IPF)	Yes	Interviews	0	5	0
				Focus groups	0	15	0
Saketkoo et al [32]	2014a	Interstitial lung disease (CTD)	Yes	Focus groups	0	45	0
				Delphi	254	0	0
Saketkoo et al [33]	2014b	Interstitial lung disease (CTD)	Yes	Nominal group technique	23	5	0
Tierney et al [34]	2013	Cleft palate, otitis media	Yes	Interviews	0	0	43
Tierney et al [35]¥	2015	Cleft palate, otitis media	Yes	Interviews	0	22	43π
Harman et al [36]	2015	Cleft palate, otitis media	Yes	Survey	0	8	35
				Delphi	104	0	0
				Consensus meeting	11	0	5
				Follow-up workshop	1	0	9
Turk et al [37]	2008	Chronic pain	Yes	Focus groups	0	31	0
				Survey	0	959	0
Core outcome set development exercises using other methods
Bellm et al [38]	2002	Oral mucositis	Not stated	OtherΩ	9	2	0
Bennett et al [39]	2012	Gestational diabetes mellitus	Not stated	Survey	4	0	2
				Consensus meeting	4	0	2
				Delphi	7	0	2
				Online evaluation	Not stated	0	Not stated
Broder et al [40]	2000	Uterine fibroids	Not stated	Delphi	9	0	1
				Nominal group technique	10	0	1
Buch et al [41]	2015	Rheumatic diseases	Not stated	OtherΩ	20	2	0
				Delphi	20	1	0
Carlson et al [42]	2003	Mania/Bipolar disorder	Not stated	Consensus conference	53	0	Not stated
Chiarotto et al [43]	2015	Lower back pain	Exempt	Delphi	129	14	0
Chitnis et al [44]	2012	Multiple Sclerosis	Not stated	Survey	51	0	Not stated
Chitnis et al [45]	2013	Multiple Sclerosis	Not stated	Consensus meeting	69	0	Not stated
				Survey	Not stated	0	0
Devane et al [5]	2007	Maternity care	Yes	Delphi	194	9	15
Gladman et al [46]	2005	Psoriatic arthritis	Not stated	Nominal group technique	Not stated	Not stated	0
Gladman et al [47]	2007	Psoriatic arthritis	Not stated	Consensus meeting	Not stated*	Not stated*	0
Goldhahn et al [48]	2014	Distal radius fracture	Not stated	Nominal group technique	Not stated0	Not stated0	0
Gonzalez et al [49]	2011	Vitiligo	Not stated	Consensus meeting	Not stated	Not stated	0
Eleftheriadou et al [50]	2012	Vitiligo	Yes	Survey	Not stated	Not statedΨ	0
Eleftheriadou et al [51]	2015	Vitiligo	Not stated	Delphi	69	32	0
Haeusler et al [52]	2015	Febrile neutropenia	Not stated	Delphi	39	4	0
Haywood et al [53]	2014	Hip fracture	Not stated	Survey	13	1	3
				Nominal group technique	22	0	3
Karas [54]	2015	Acute diarrhoea	Not stated	Delphi	70	0	31
Katona et al [55]	2007	Dementia	Not stated	Consensus conference	33	0	3
Merkies et al [56]	2006	Peripheral neuropathy	Not stated	Consensus conference	22	1	0
Moniz-Cook et al [57]	2008	Dementia care	Not stated	Consensus meeting	Not stated	0	0
				Survey	131	0	5
				Consensus meeting	Not stated	0	0
				Consensus meeting	Not stated	0	0
Reilly et al [58]	2006	Charcot-Marie-Tooth disease type 1A	Not stated	Consensus conference	21	2	0
Salaffi et al [59]	2012	Fibromyalgia	Yes	Delphi	252	86	0
Schmitt et al [60]	2007	Eczema	Not stated	Survey	6	4	2
Schmitt et al [61]	2010	Eczema	Not stated	Consensus conference	Not stated#	Not stated#	Not stated#
Schmitt et al [62]	2011	Eczema	Not stated	Delphi	40	6	0
Schmitt et al [63]	2012	Eczema	Not stated	Nominal group technique	38	5	0
Sinha et al [64]	2012	Asthma	Yes	Delphi	46	0	0
				Survey	0	11	88
Stuart et al [65]	2011	Ovarian cancer	Not stated	Consensus conference	Not stated	0	Not stated
Thigpen et al [66]	2011	Ovarian cancer	Not stated	Consensus conference	Not stated	0	Not stated
Tugwell et al [67]	1993	Rheumatoid arthritis	Not stated	Consensus conference	Not stated	Not stated	Not stated
				Nominal group technique	Not stated	Not stated	Not stated
Kirwan et al [68]	2003	Rheumatoid arthritis	Not stated	Consensus conference	46	11	0
Kirwan et al [69]	2005	Rheumatoid arthritis	Not stated	Consensus conference	160	19	0
Kirwan et al [70]	2007	Rheumatoid arthritis	Not stated	Consensus conference	60	20	0
Van der Heijde et al [71]	1997	Ankylosing spondylitis	Not stated	OtherΩ	Not stated	Not stated	0
				Nominal group technique	Not stated	Not stated	0
Vargus-Adams et al [72]	2009	Cerebral palsy	Yes	Delphi	39	21	23
MacKichen et al [18]α	2015	Chronic pain after total knee replacement	Not stated	Focus groups	18	0	0
Wylde et el [73]	2015	Chronic pain after total knee replacement	Yes	Delphi	39	71	0
				Consensus meeting	Not stated	12	0

*total number of participants = 137, breakdown not provided.

⁰ total number of participants = 26, breakdown not provided.

^#total number of participants = 40, breakdown not provided.

^Ψ total number of participants = 461, breakdown not provided.

^α Papers not in original search, included because they are part of the COS pathway.

^¥ Full details given in Tierney 2013.

^Ω Other = meetings/semi-structured discussions.

Papers in italics have a qualitative component which was carried out prior to the COS exercise and was not specifically designed for the COS work but has fed into it.

Not Stated = participants in stakeholder group implied by numbers not given.

N.B. Papers grouped together are part of the same COS exercise.

The data collection methods used in COS development including patients and carers were; Delphi exercises (n = 14), consensus conferences/meetings (n = 20), surveys (n = 10), interviews (n = 6), focus groups (n = 6), nominal group techniques (n = 9) and other (n = 5). For the methods where participant numbers were reported fully, the mean percentage of patients and carers in each ranged from 20% to 86% (Table 2). Patients and carers formed the majority or all of the participants in surveys, interviews and focus groups. Health Professionals formed the majority of participants in nominal group techniques, Delphi exercises and consensus meetings. In all cases where qualitative data collection methods were utilised the primary aim was to identify outcomes of importance to the participants.

Table 2. Mean percentage of Health professionals to patients and carers by data collection method.

Method	Health Professionals (mean %)	Patients and carers (mean %)
Consensus meetings/conferences (n = 9)	80	20
Delphi (n = 19)	77	23
Nominal group technique (n = 5)	70	30
Surveys (n = 8)	36	64
Interviews (n = 7)	22	78
Focus groups (n = 7)	14	86

N.B. not all papers provided a full breakdown of participants and are therefore not included.

Qualitative studies to inform COS development involving patients and carers

Ten papers [16, 19, 21, 22, 24, 30, 32, 34, 35, 37] reported using qualitative data collection methods to identify outcomes important to patients and carers (Table 3). However, of these only three were clearly reported as part of a COS development process [16, 24, 30]; five discussed outcomes with patients and carers but it was unclear whether the data were collected specifically with the intention to include them in a COS [21, 32, 34, 35, 37]; and two further studies were conducted for other primary research aims, such as exploring perceptions of access to care [19, 22].

Table 3. Reporting of qualitative methods with patients and carers.

	Allard 2014 [16]	Arnold 2008 [24]	Potter 2013 [19]	Saketkoo 2014a [32]	Swigris 2005 [22]	Stamm 2009 [21]	Sanderson 2010a [30]	Tierney 2013 [34]	Tierney 2015 [35]	Turk 2008 [37]
Health area	Neurodisability	Fibromyalgia	Breast cancer	Interstitial lung disease	Interstitial lung disease	Osteoarthritis	Rheumatoid arthritis	Cleft palate, otitis media	Cleft palate, otitis media	Chronic pain
Theoretical framework	N/R	N/R	N/R	N/R	N/R	N/R	N/R	N/R	N/R	N/R
Sampling
Approach	Purposive	Purposive	Purposive	N/R#	Purposive	Purposive	Purposive	Purposive	Purposive	N/R#
No. approached	N/R	N/R	N/R	N/R	N/R	N/R	N/R	N/R	N/R	N/R
No. taking part	107	48	31	45	20	56	23	43	22	31
Age	Yes	Yes	Yes	YesΨ	Yes	Yes	Yes	Yes	Yes	Yes
Gender	Yes	Yes	Yes	YesΨ	Yes	Yes	Yes	Yes	Yes	Yes
Ethnicity	Yes	Yes	N/R	YesΨ	N/R	N/R	N/R	N/R	N/R	Yes
Socio economic status	Yes	N/R	N/R	N/R	N/R	N/R	N/R	N/R	N/R	Yes
Clinically relevant info	Yes	Yes	Yes	YesΨ	Yes	Yes	Yes	Yes	Yes	Yes
Data collection
Focus groups	Yes	Yes	n/a	Yes	Yes	Yes	n/a	n/a	n/a	Yes
Interviews	Yes	n/a	Yes	n/a	Yes	n/a	Yes	Yes	Yes	n/a
Data analysis
A priori categories applied	Yes (plus emergent themes)	No	No	No	No	No	No	No	No	No
Thematic/content analysis*	Yes	n/a	n/a	Yes	Yes	Yes	n/a	Yes	Yes	Yes
Grounded theory**	No	Yes	Yes	No	No	No	Yes	No	No	No
Framework approach***	Yes	No	No	No	No	No	Yes	Yes	Yes	No
Data saturation mentioned	Yes	No	Yes	No	Yes	No	Yes	No	No	No
Triangulation
Multiple coders/perspectives	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Data collection methods (both focus groups and interviews)	Yes	No	No	No	Yes	No	No	No	Yes	No
Reflexivity	No	Yes	Yes	No	No	No	No	No	No	No

N/R = not reported, n/a = not applicable,

^# = Some indication of types of participant included but not the sampling approach used.

^Ψ Details reported in Saketkoo 2014b.

* Paper describes a generic thematic / content approach

** Paper describes analysis informed by Grounded Theory approaches, rather than an explicit Grounded Theory methodological framework

*** Paper refers to the use of framework as part of the analytical process

Methodological framework

Of the papers reporting a qualitative approach in the development of a COS [16, 19, 21, 22, 24, 30, 32, 34, 35, 37] none explicitly reported an underpinning methodological framework.

Sampling

Eight papers [16, 19, 21, 22, 24, 30, 34, 35] reported using purposive sampling and two [32, 37] did not discuss a clear sampling strategy. All papers gave details of participant age, gender and some clinical detail (e.g. disease severity). However, some omitted details on socio economic status [19, 21, 22, 24, 30, 32, 34, 35] and ethnicity [19, 21, 22, 30, 34, 35]. Nine papers [16, 19, 22, 24, 30, 32, 34, 35, 37] recruited participants from a single country but one paper [21] included participants from five different European countries.

Data collection

Four papers reported interviews with patients and carers as the only data collection method [19, 30, 34, 35]. The number of interviews reported in these studies ranged from 22 to 31. Focus groups were the only method reported in four papers: Arnold [24] and Saketkoo [32] conducted six focus groups, Turk [37] four, and Stamm [21] ten. There was an average of seven participants per group. A combination of interviews and focus groups were reported in two papers; Allard [16] who carried out ten individual interviews and 12 focus groups (97 participants) and Swigris [22] who conducted five individual interviews and three focus groups (15 participants).

All studies were audio-recorded and transcribed verbatim, and all reported using topic or discussion guides. However, only four [16, 19, 30, 34] provided details of the contents or derivation of these.

Data analysis

Only one of the papers [16] reported the use of a priori categories in the analysis, with most using exclusively inductive data-driven approaches. Seven papers [16, 21, 22, 32, 34, 35, 37] reported using thematic or content analysis. Arnold [24], Potter [19] and Sanderson [30] reported analysis based upon the principles of grounded theory. Four papers also referred to the use of the framework approach in the analytical description [16, 30, 34, 35].

Four interview studies reported reaching data saturation [16, 19, 22, 30]. In two cases, saturation was judged in conjunction with focus group data [16, 22]. None of the papers reporting on focus groups only [21, 24, 32, 37] described achieving data saturation.

Triangulation

Data were analysed using multiple coders of and/or perspectives on the data in all of the included papers. Three [16, 22, 35] reported on the triangulation [74] of two different data collection methods (interviews and focus groups).

Reflexivity

Only two papers [19, 24] included any reflexive content. Potter [19] reported the use of a medically trained interviewer and reflected on the influence this may have had on the results. Arnold [24] discussed the use of an experienced facilitator with no prior knowledge of the condition under investigation, to avoid leading discussions.

Reported strengths and limitations

All studies acknowledged some strengths and limitations of their work. The main limitation noted in the included papers concerned recruitment. Five papers reported on the difficulty in recruiting an ethnically diverse and gender balanced sample [16, 24, 30, 33, 34]. Four discussed that they were unable to recruit participants to fulfil the desired sampling quota, for example, with not all categories of the disease/investigation under investigation being included [16, 24, 30, 37].

Sanderson [31] and Turk [37] reported that patients and carers identified a rather large and varied range of outcomes important to them, making subsequent rating and ranking very difficult. Potter [19] and Tierney [34] also highlighted the difficulty that participants may have when asked to recall their treatments and experience.

The participation of patients and carers in the core outcome set process was identified as a strength in 3 papers. Allard [16] stated that the differences between patients and parents were highlighted, and Sanderson [30] and Saketkoo [32] both reported that outcomes identified by patients and carers as important to them were not in current professional core outcome sets.

Resource use associated with qualitative methodology

Other than details on the length of the focus groups and interviews and reimbursements, very little information about the resources required to carry out qualitative data collection methods in COS development was reported in the included papers.

Planned and ongoing studies

As of 13 August 2015, 73 studies (S1 appendix) were registered on the COMET database as planned or ongoing COS development studies. Of these, all studies reported that they intended to include patients and carers as participants and 37 (52%) stated they wouldbe using qualitative methods as part of the COS pathway (S2 appendix). Of these 37, 21 planned to use both interviews and focus groups. Of the planned and ongoing studies five have published protocols [75–79]. Overall this supports the findings in Gorst’s [80] recently updated systematic review which reports the increase in COS development studies and the increasing involvement of patients and carers.

Discussion

This study has described the different data collection methods used by COS developers in order to elicit patient and carer views on their preferred treatment outcomes, and has focused specifically on the reported use of methodology and methods within associated qualitative research. To our knowledge it is the first review to focus specifically on the use and reporting of qualitative research in COS development to date.

We have used the COMET database to identify COS studies. The database is based upon a systematic review of relevant studies and is regularly updated to ensure that new studies are added as they become available [4]. However, it is possible that there may be a lag time before studies are added to the database, or that COS relevant studies are not indexed or reported in ways that would mean they are captured within the database. We did perform reference searches of papers identified and found additional studies via this method. Therefore whilst the COMET database is an appropriate source of COS studies based on systematic review methodology, there is a possibility that there may be additional relevant studies not captured here. Furthermore, the focus of this review has been on participation in COS exercises. That is, we were concerned with the participation of patients and carers as research participants contributing data to the development of COSs. Of course patients and carers can also contribute to COS development via involvement in the research process as research partners and advisors, and in doing so influence the research and outputs. Our review has not focused on this involvement, which may not always be well reported and detailed in the outputs of COS studies. Further work to examine patient and carer involvement and its contribution to COS development would be useful. Recent work focused on patient and carer involvement demonstrates that this is a key issue that should be considered by researchers in the field [81].

To date, patients have participated in exercises that both identify relevant outcome domains for consideration in a COS, such as interviews, focus groups, and surveys, and also in the prioritisation and consensus methods that finalise a COS. However, the number and types of participants taking part in these, particularly patients and carers, were sometimes difficult to discern from the papers included in this review. Where participant numbers were reported, proportionately more patients participated in methods designed to identify relevant outcome domains for consideration, whilst Health Professionals were represented more than patients in prioritisation and consensus methods.

In this sample, qualitative and survey approaches have included patients more than Health Professionals to identify outcome domains. These methods may help incorporate patient perspectives that might otherwise go unheard, via the inclusion of patient preferred outcome domains in subsequent prioritisation and consensus methods. However, in prioritisation exercises (primarily Delphi methods in COS to date) where participant perspectives are aggregated and quantified then absolute numbers of participants from different stakeholder groups will influence outputs, particularly where views differ substantially between patients and carers and other stakeholder groups. Our observation that patients appear to be in the minority in these methods should be cause for reflection, although even if the number of participants from different stakeholder groups were balanced, there is some evidence to suggest that patients and carers rate many or all outcome domains as important in such exercises [31, 37]. If this were the case then other stakeholder views may dominate as the outcome domains they do not value, on aggregate, will not be taken forward to the final COS. The inclusion of qualitative research to incorporate patient and carer perspectives as a precursor to group consensus approaches will not necessarily guarantee that patients’ views are taken forward to the final COS.

In consensus meetings and conferences, one might expect Health Professionals to have more representation, as demonstrated here. The impact of patients and carers on outputs in these circumstances may well depend on the process and means by which their views are facilitated and accommodated, and on who takes part [81], as much as absolute numbers present versus other stakeholder groups.

Interestingly, one of the COS exercises we identified was expressly focused on understanding patient views and developing a patient core outcome set [30], building on the OMERACT work in rheumatoid arthritis. This approach does not rely on the integration of Health Professional and patient and carer views in a single COS exercise. Rather, it explicitly acknowledges that patient views are likely to be different to those of other stakeholders and therefore need specific consideration.

All of the qualitative research reviewed here was utilised as a means to ensure that patient and carer perspectives on outcome domains were accommodated in COS development processes. However, a key observation from this review is that some of this research (2 out of 10 papers) appear to have been designed and conducted for other primary research aims, not associated with COS development. Exceptions to this are Sanderson [30], Allard [16] and Arnold [24]. The remaining five papers [21, 32, 34, 35, 37] discussed outcomes with patients and carers but it is unclear whether the information was collected specifically with the intention to include it in a COS exercise. The availability of related research which is ready to feed into COS development, and the desire to include patient perspectives when specific COS-focused primary qualitative work is potentially time and resource intensive, could explain this. However, it does raise questions about the precise applicability of the underpinning qualitative research.

None of the papers stated a clear overarching qualitative methodological framework. Three [19, 24, 30] mentioned Grounded Theory, but only in descriptions of analytical approach. This may well be perfectly justifiable, for example if COS related qualitative research is being undertaken from an overtly generic qualitative research standpoint [82, 83], though to date there does not appear to have been any reflection on this in the primary COS papers, or the COS methodological literature. There are longstanding debates within the qualitative methodological literature [82, 83] about the use of methodological frameworks (e.g. grounded theory, phenomenology, ethnography), the coherency of underpinning research methods, and suitability for specific research aims. These have generally been in response to generic qualitative approaches that mix and match research methods, and which are more common in health-related research.

Reflection on methodology may help to further define the purpose and role of qualitative research in COS development. For example, if the primary aim is to understand and explain patients’ perspectives as a means to influence COS development then an approach such as Grounded Theory may be appropriate; if one argues that the lived experience of illness and treatment is of chief concern as it describes the essence of disease experience which we are trying to improve via trial research then a phenomenological framework may be informative; or if we are simply wanting a pragmatic precursor that lists ‘patient priorities’ from descriptive accounts prior to COS prioritisation exercises, then forms of generic descriptive qualitative research may suffice. Whilst to our knowledge these issues have not been considered in COS research, suitable methodological frameworks have been discussed in analogous outcome related work in the patient reported outcome development literature concerning content validity [84]. It would seem that there are very clear parallels between this work and COS qualitative research.

Thinking about data collection methods specifically, this review demonstrates the use of interviews, focus groups, and a combination of these. Whilst there has been some recent attention to this in the methodological literature [85] empirical comparisons of the outputs and relative merits of different methods is needed in this developing area. Some of this work is currently underway (S3 appendix).

The qualitative papers reviewed note limitations around recruitment and diversity of samples. The reporting of sample characteristics is also variable. This is important when considering the generalisability (transferability in qualitative terms) of findings. Implicitly, COS development for trials assumes the generalisability of the final COS to varied populations and settings [81]. Conversely, qualitative approaches may not necessarily lay claim to widespread generalisability, acknowledging that patients’ and carers’ perspectives may vary, for example, in time, place and according to cultural factors. In addition to clear reporting of sample characteristics to aid judgements of transferability this issue needs acknowledgment. This has been addressed in some COS work, for example with work to culturally validate a patient core set amongst non-white patients in rheumatoid arthritis [86]. Outcomes and their importance in health research is often a difficult concept for patients and carers to understand [16, 85]. The data collection methods used may have a direct impact on the depth of explanation of outcomes required [81]. For example, qualitative methods allow participants to talk about their experiences of illness without the need for an in-depth understanding of outcomes [87]. There is currently no guidance on how to discuss outcomes with patients and carers in qualitative research. The sharing of best practice and the publication of topic guides will aid future COS work [81, 85].

There are some fairly simple reporting recommendations (Table 4) that we would make for future qualitative COS work which include: clear reporting of aims in relation to the COS development; sampling and sample characteristics; data collection methods and derivation; use and reporting of the topic guide; data analysis; overt description of findings in the context of the COS; and reflection on strengths and limitations of approach. Beyond this we would suggest that there is a need for more fundamental consideration of the role of qualitative methods in COS and related methodological approaches, of the relative merits of different data collection approaches in terms of knowledge produced and time and resource requirements, as well as claims to generalisability.

Table 4. Reporting recommendations for qualitative research methods in COS development.

1.	Research aims and relationship with broader COS development process
2.	Sampling approach
3.	Type of data collection methods (e.g. interviews, focus groups, combination); content and derivation / justification (e.g. topic guide)
4.	Analytical approach and justification
5.	Sample characteristics and participants numbers
6.	Findings related to outcome domains (concordant with research aims)
7.	Report approaches to ensuring rigour (e.g. multiple perspectives on the data, respondent validation) and consider reflexive content
8.	Discuss strengths and limitations of approach

Supporting information

S1 Appendix. Planned and ongoing studies.

(DOCX)

S2 Appendix. Planned and ongoing studies using qualitative methods.

(DOCX)

S3 Appendix. Unpublished work.

(DOCX)

S4 Appendix. PRISMA statement.

(DOC)

Data Availability

All relevant data are within the paper and its Supporting Information files.

Funding Statement

This work was funded by a College of Medical and Dental Sciences PhD studentship, University of Birmingham.