Abstract
Background
The Core Outcome Measures in Effectiveness Trials (COMET) database is a publically available, searchable repository of published and ongoing core outcome set (COS) studies. An annual systematic review update is carried out to maintain the currency of database content.
Methods
The methods used in the fourth update of the systematic review followed the same approach used in the original review and previous updates. Studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. Searches were carried out in March 2018 to identify studies that had been published or indexed between January 2017 and the end of December 2017.
Results
Forty-eight new studies, describing the development of 56 COS, were included. There has been an increase in the number of studies clearly specifying the scope of the COS in terms of the population (n = 43, 90%) and intervention (n = 48, 100%) characteristics. Public participation has continued to rise with over half (n = 27, 56%) of studies in the current review including input from members of the public. The rate of inclusion of all stakeholder groups has increased, in particular participation from non-clinical research experts has risen from 32% (mean average in previous reviews) to 62% (n = 29). Input from participants located in Australasia (n = 17; 41%), Asia (n = 18; 44%), South America (n = 13; 32%) and Africa (n = 7; 17%) have all increased since the previous reviews.
Conclusion
This update included a pronounced increase in the number of new COS identified compared to the previous three updates. There was an improvement in the reporting of the scope, stakeholder participants and methods used. Furthermore, there has been an increase in participation from Australasia, Asia, South America and Africa. These advancements are reflective of the efforts made in recent years to raise awareness about the need for COS development and uptake, as well as developments in COS methodology.
Introduction
Insufficient consideration of the outcomes measured in clinical trials, and other studies, is increasingly being addressed through the development and use of core outcome sets (COS). A COS is an agreed standardised set of outcomes that should be measured and reported, as a minimum, in all trials for a specific clinical area [1]. The Core Outcome Measures in Effectiveness Trials (COMET) Initiative aims to collate and stimulate the development and application of COS, by maintaining a public repository of studies relevant to the development of COS (The COMET database, http://www.comet-initiative.org/studies/search). A survey of users of the database was undertaken to understand the reasons why people were searching the COMET database [2]. The most frequent users were people thinking about developing a COS to see whether a COS already exists in their area of interest to avoid any duplication, therefore emphasising the importance of keeping the database current.
A systematic review was conducted to initially populate the COMET database [3], and it has been subsequently updated annually to include all published COS up to, and including, 2016 [2, 4, 5]. The previous update showed that COS developers are adopting a more structured approach towards COS development, and public participation in COS development has also increased [5]. The review, however, highlighted a gap in the involvement of stakeholders from a wider range of geographical settings, in particular low and middle income countries (LMIC). This is important to increase the applicability of COS to global health and tackling the global burden of disease [6].
The database is therefore an integral resource to not only the development of COS, but also to the uptake of COS in research and in the avoidance of unnecessary duplication and waste of scarce resources. Eligible studies are added to the database as they are found; however, an additional annual update to the systematic review means that the database is kept up to date.
The aims of the current study were consistent with the previous update [5], specifically to: (i) update the systematic review in order to identify any further studies where a COS has been developed; (ii) to describe the methodological approaches taken in these studies, and (iii) to highlight areas for future COS development and improvement.
Methods
Systematic review update
The methods used in this updated review followed the same approach used in the original review [3] and previous updates [2, 4, 5]. The methods were described in detail previously [3, 5]; a summary of methods is provided below.
Study selection
The inclusion and exclusion criteria were described in detail in the original systematic review [3]. Studies were eligible for inclusion if they had applied methodology for determining which outcome domains or outcomes should be measured in clinical trials or other forms of health research. As described previously, studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. Studies describing the development of a Patient Reported Outcome (PRO) COS (a core set of patient-reported symptoms and health related quality of life domains) or Core Event Set (a core set of adverse events or complications) were eligible for inclusion in the review update. Studies describing the update of an existing COS were included as linked papers to the original COS.
Identification of relevant studies
In March 2018, MEDLINE via Ovid and SCOPUS were searched without language restrictions. The search identified studies that had been published or indexed since the previous systematic review update, between January 2017 and the end of December 2017. The multifaceted search strategy, developed for the original review using a combination of text words and index terms, was used for the current update [3] (S1 Table). Ovid included a new search option entitled ‘MEDALL’ that incorporates all previous search options and provides the most extensive coverage, including the ‘in-process’ citations as well as E-pub ahead of print citations. This search option was selected for this update of the systematic review in order to identify eligible studies at the earliest possible opportunity.
Hand searching was completed, including any studies that had been submitted directly to the COMET database, references cited in eligible studies, as well as those in ineligible studies that referred to or used a COS.
Selecting studies for inclusion in the review
As described previously [3, 5], records from each database were combined and duplicates removed. Titles and abstracts were read to assess eligibility of studies for inclusion in the review (stage 1). Full texts of potentially relevant articles were obtained to assess for inclusion (stage 2). Two of five reviewers (EG, SG, NH, VS and KMS) independently checked the title and abstract of each citation. Citations were retained for further checking if agreement could not be reached. Two of the same five reviewers assessed each full paper for inclusion in the review. In cases of disagreement a third reviewer was consulted. The reasons for exclusion at this stage were documented for articles judged to be ineligible.
Checking for agreement between reviewers
One reviewer was new to this systematic review and had not been involved in previous updates (KMS). During each stage of the review process, agreement between KMS and the lead reviewer (EG) was assessed prior to independently assessing records. Agreement between all other reviewers had been checked in previous updates.
Checking for correct exclusion
At abstract stage
A 1% sample of the records excluded on the basis of the title and abstract was checked by a sixth reviewer (Jamie Kirkham) and assessed for correct exclusion. If any studies were identified as being incorrectly excluded, further checking was performed within the other excluded records.
At full text stage
Of the records that had been excluded after reading the full text, 5% were assessed for correct exclusion. If any studies were identified as being incorrectly excluded at this stage, further checking was performed.
Data extraction
Described in full previously [3], data were extracted in relation to the study aim(s), health area, target population, interventions covered, methods of COS development and stakeholder groups involved. Data relating to the geographical locations of participants included in the development of COS were also extracted.
Data analysis and presentation of results
We report the review in accordance with PRISMA guidelines [7] (S1 File). We describe the studies narratively, and present the findings in text and tables. As in previous updates, we did not anticipate conducting any statistical analyses to combine the findings.
Results
Description of studies
Following the removal of duplicates, 5140 records were identified in the database search. A total of 4626 records were excluded during the title and abstract stage, and a further 444 were excluded following the assessment of full text (Fig 1). Table 1 provides a summary of the reasons for exclusion of records at full text stage. Seventy records related to 45 new studies that met the inclusion criteria. In addition to the review search, three additional records were identified through database search alerts as being eligible for inclusion in the review. These three studies were not identified during the review search, as although they were published in 2017, they had not been indexed in the databases at the time we ran our search. A further 15 reports were identified by hand searching references of included studies. In total, 88 reports relating to 48 new studies describing the development of 56 COS were included for the first time in this update (S2 Table).
Fig 1. Identification of studies.
Table 1. Reasons for exclusion at full text stage.
| Exclusion Categories of Full Text Stage | Number of records |
|---|---|
| Studies relating to how, rather than which, outcomes should be measured | 50 |
| Studies reporting the design/ rationale of single trial | 3 |
| Studies reporting the use of a COS | 1 |
| Systematic reviews of clinical trials | 39 |
| Review/overview/discussion only, no outcome recommendations | 113 |
| Core outcomes/ outcome recommendations not made | 72 |
| Quality indicators | 5 |
| One outcome/ domain only | 8 |
| Instrument development | 8 |
| Recommendations by single author only | 2 |
| ICF Core set | 1 |
| Preclinical/ Early phase only (0, I, II) | 8 |
| Irrelevant | 43 |
| Assessed in previous review | 3 |
| ICF Core set development | 2 |
| HRQL | 3 |
| Recommendations for clinical management in practice not research | 61 |
| Studies that elicit stakeholder group opinion regarding which outcome domains or outcomes are important | 8 |
| Ongoing studies | 14 |
Included studies
Year of publication
The year of first publication of COS has been updated to include the 48 new studies included in this update (Fig 2). Of the 48 studies identified in this update, 36 were published in 2017, 11 were published in 2016 and one study was published in 2015.
Fig 2. Year of first publication of each COS study (n = 307).
Scope of core outcome sets
The scope of published COS studies is summarised in Table 2. This includes study aims, setting for intended use, population characteristics and intervention characteristics. Table 2 highlights that there has been an increase in the number of COS studies specifying the population (n = 43, 90%) and intervention (n = 48, 100%) characteristics of the newly developed COS. Fig 3 displays the number of COS that have been developed in each disease category. The majority of COS have been developed within the areas of cancer, rheumatology, heart & circulation, neurology and orthopaedics & trauma. Disease categories and disease names are provided in S2 Table.
Table 2. The scope of included studies (n = 307).
| Original review n (%) |
Update review 1 n (%) |
Update review 2 n (%) |
Update review 3 n (%) | Update review 4 n (%) | Combined* N (%) |
|
|---|---|---|---|---|---|---|
| Study aims | ||||||
| Specifically considered outcome selection and measurement | 98 (50) | 21 (75) | 13 (65) | 10 (60) | 33 (69) | 175 (57) |
| Considered outcomes while addressing wider clinical trial design issues | 98 (50) | 7 (25) | 7 (35) | 5 (40) | 15 (31) | 132 (43) |
| Intended use of recommendations | ||||||
| Clinical research | 174 (89) | 24 (86) | 19 (95) | 9 (60) | 44 (92) | 270 (88) |
| Clinical research and practice | 22 (11) | 4 (14) | 1 (5) | 6 (40) | 4 (8) | 37 (12) |
| Population characteristics | ||||||
| Adults | 12 (6) | 12 (43) | 5 (25) | 10 (67) | 21 (44) | 63 (21) |
| Children | 22 (11) | 2 (7) | 6 (30) | 0 (0) | 5 (10) | 35 (11) |
| Adults and children | 12 (6) | 2 (7) | 0 (0) | 3 (20) | 10 (21) | 27 (9) |
| Older adults | 2 (1) | 1 (4) | 0 (0) | 0 (0) | 3 (6) | 6 (2) |
| Adolescents and adults | 0 (0) | 0 (0) | 0 (0) | 1 (7) | 4 (8) | 5 (2) |
| Not specified | 148 (76) | 11 (39) | 9 (45) | 2 (13) | 5 (10) | 171 (56) |
| Intervention characteristics | ||||||
| All intervention types | 7 (4) | 8 (29) | 12 (60) | 8 (53) | 29 (60) | 68 (22) |
| Drug treatments | 39 (20) | 4 (14) | 0 (0) | 0 (0) | 4 (8) | 46 (15) |
| Surgery | 13 (7) | 4 (14) | 4 (20) | 4 (27) | 7 (15) | 32 (10) |
| Vaccine | 2 (1) | 0 (0) | 0 (0) | 0 (0) | 0 | 2 (1) |
| Rehabilitation | 1 (1) | 1 (4) | 0 (0) | 1 (7) | 2 (4) | 5 (2) |
| Exercise | 1 (1) | 1 (4) | 1 (5) | 0 (0) | 1 (2) | 4 (1) |
| Exercise (physical activity) | 1 | 0 | 1 | 0 | 1 | |
| Exercise (yoga) | 0 | 1 | 0 | 0 | 0 | |
| Procedure | 4 (2) | 0 (0) | 2 (10) | 0 (0) | 2 (4) | 8 (3) |
| Device | 3 (2) | 0 (0) | 0 (0) | 1 (7) | 0 | 4 (1) |
| Other | 11 (6) | 5 (18) | 0 (0) | 1 (7) | 3 (6) | 20 (7) |
| Not specified | 115 (59) | 5 (18) | 1 (5) | 0 (0) | 0 | 118 (38) |
*Additional information provided by updated papers linked to previously published COS are reflected in the combined column.
Fig 3. Number of COS developed in each disease category (n = 307).
Methods used to select outcomes
The methods used to develop the 48 new COS identified in the current review are presented in Table 3 alongside the methods used in the four previous systematic reviews. Table 3 highlights that the use of mixed methods to develop a COS has remained high (n = 35; 73%). The use of the Delphi technique has also remained high with 48% (n = 23) of studies using the technique in combination with other methods.
Table 3. The methods used to develop COS (n = 307).
| Main methods | Original review n (%) | Update review 1 n (%) | Update review 2 n (%) | Update review 3 n (%) | Update review 4 n (%) |
Combined* N (%) |
|---|---|---|---|---|---|---|
| Semi-structured group discussion only | 55 (28) | 2 (7) | 2 (10) | 0 (0) | 3 (6) | 62 (20) |
| Unstructured group discussion only | 18 (9) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 18 (6) |
| Consensus development conference only | 12 (6) | 0 (0) | 1 (5) | 0 (0) | 1 (2) | 14 (5) |
| Literature/systematic review only | 11 (6) | 5 (18) | 2 (10) | 1 (7) | 6 (13) | 25 (8) |
| Delphi only | 6 (3) | 2 (7) | 2 (10) | 0 (0) | 0 (0) | 10 (3) |
| Survey only | 3 (2) | 0 (0) | 0 (0) | 0 (0) | 1 (2) | 4 (1) |
| NGT only | 1 (1) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 1 (<1) |
| Interview only | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 1 (2) | 1 (<1) |
| Mixed methods (see descriptions below) | 74 (38) | 17 (61) | 13 (65) | 12 (80) | 35 (73) | 151 (49) |
| Delphi + another method(s) | 22 (11) | 6 (21) | 9 (45) | 9 (60) | 23 (48) | 71 (23) |
| Semi-structured group discussion + another method(s) | 30 (15) | 7 (25) | 4 (20) | 2 (13) | 9 (19) | 51 (17) |
| Consensus development conference + another method(s) | 7 (4) | 0 (0) | 0 (0) | 0 (0) | 1 (2) | 8 (3) |
| Literature/systematic review + another method(s) | 10 (5) | 4 (14) | 0 (0) | 1 (7) | 2 (4) | 16 (5) |
| NGT + another method(s) | 4 (2) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 4 (1) |
| Focus group + another method(s) | 1 (1) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 1 (<1) |
| No methods described | 16 (8) | 2 (7) | 0 (0) | 2 (13) | 1 (2) | 21 (7) |
*Additional information provided by updated papers linked to previously published COS are reflected in the combined column.
People involved in selecting outcomes
The list of stakeholders included in COS development (Table 4) has been updated to include the COS identified in this update. Of the 307 published COS studies, 273 (89%) provided information about the stakeholders who participated in the COS development. Clinical experts were included in 268 (98%) studies; this is in direct contrast to public representatives who were included in 89 (33%) studies. Public participation has continued to rise in recent years, with 56% (n = 27) of studies in the current review including input from members of the public. The rate of inclusion of all stakeholder groups has increased in comparison to the original review and previous review updates, with participation from non-clinical research experts, in particular, rising from 32% (n = 73) as the mean average in previous reviews to 62% (n = 29) in the current review.
Table 4. Participant groups involved in selecting outcomes for inclusion in COS (n = 307).
| Participants category | Sub-category (not mutually exclusive) | Frequency of participants | |||||
|---|---|---|---|---|---|---|---|
| Original review n (%) |
Update review 1 n (%) |
Update review 2 n (%) |
Update review 3 n (%) |
Update review 4 n (%) |
Combined^ N (%) |
||
| Clinical experts | 171 (99) | 20 (95) | 17 (100) | 14 (93) | 46 (96) | 268 (98) | |
| Clinical experts | 86 | 14 | 16 | 14 | 38 (79) | 169 (62) | |
| Clinical research expertise | 66 | 9 | 9 | 2 | 26 (54) | 114 (42) | |
| Clinical trialists/ Members of a clinical trial network | 9 | 2 | 1 (2) | 12 (4) | |||
| Others with assumptions* | 54 | 54 (20) | |||||
| Public representatives | 30 (17) | 13 (62) | 11 (65) | 8 (53) | 27 (56) | 89 (33) | |
| Patients | 19 | 11 | 7 | 8 | 18 (38) | 63 (23) | |
| Carers | 7 | 1 | 3 | 3 | 9 (19) | 23 (8) | |
| Patient support group representatives | 9 | 1 | 4 | 9 (19) | 24 (9) | ||
| Service users | 2 | 1 | 2 (4) | 5 (2) | |||
| Non-clinical research experts | 53 (31) | 9 (43) | 9 (53) | 2 (13) | 29 (60) | 103 (38) | |
| Researchers | 26 | 4 | 4 | 2 | 26 (54) | 64 (23) | |
| Statisticians | 19 | 4 | 3 | 1 (2) | 27 (10) | ||
| Epidemiologists | 11 | 2 | 1 | 4 (8) | 19 (7) | ||
| Academic research representatives | 4 | 4 (2) | |||||
| Methodologists | 6 | 3 | 2 | 4 (8) | 16 (6) | ||
| Economists | 3 | 1 | 2 (4) | 6 (2) | |||
| Authorities | 39 (23) | 5 (24) | 3 (18) | 0 (0) | 12 (25) | 59 (22) | |
| Regulatory agency representatives | 30 | 4 | 3 | 6 (13) | 43 (16) | ||
| Governmental agencies | 12 | 1 | 5 (10) | 18 (7) | |||
| Policy makers | 4 | 1 | 3 (6) | 8 (3) | |||
| Charities | 1 | 1 (2) | 2 (1) | ||||
| Service commissioners | 3 (6) | 3 (1) | |||||
| Industry representatives | 31 (18) | 4 (19) | 3 (18) | 0 (0) | 9 (19) | 47 (17) | |
| Pharmaceutical industry representatives | 28 | 3 | 3 | 8 (17) | 42 (15) | ||
| Device manufacturers | 2 | 1 | 1 (2) | 4 (2) | |||
| Biotechnology company representatives | 1 | 1 (<1) | |||||
| Others | 72 (42) | 2 (10) | 1 (6) | 1 (7) | 8 (17) | 84 (31) | |
| Service providers | 4 (8) | 4 (2) | |||||
| Ethicists | 1 | 1 (<1) | |||||
| Journal editors | 2 | 1 | 2 (4) | 5 (2) | |||
| Funding bodies | 1 | 1 (<1) | |||||
| Yoga therapists/ instructors | 1 | 1 (<1) | |||||
| Members of health care transition research consortium | 1 | 1 (<1) | |||||
| Educationalist | 1 (2) | 1 (<1) | |||||
| Nutritionist | 1 (2) | 1 (<1) | |||||
| National professional and academic bodies/ committees | 1 (2) | 1 (<1) | |||||
| Others** (besides known participants) | 15 | 15 (6) | |||||
| Others with assumptions* | 54 | 54 (20) | |||||
| No details given | 24 (12) | 7 (25) | 3 (15) | 0 (0) | 0 (0) | 34 (11) | |
* 54 studies with clinical input but unclear about involvement of other stakeholders
** Workshop/meeting participants (*5), subcommittee/committee (*2), guidelines panel, military personnel, moderator and audience, representatives from EORTC, members with expertise in information technologies, informatics, clinical registries, data-standards development, expertise in vaccine safety, malaria control and representatives from funding agencies/registration authorities, and donor organisation, members of the Rheumatology Section of the American Academy of Pediatrics, the Pediatric Section of the ACR, and the Arthritis Foundation, the diagnostic radiology and basic science communities, and from individuals conversant with functional and quality of life (QOL) assessments, comparative effectiveness research, and cost/ benefit analysis
^Additional information provided by updated papers linked to previously published COS are reflected in the combined column
As described previously [3, 5], public representatives include patients, carers, health and social care service users and people from organisations who represent these groups. The degree of public participation within the development of the COS studies included in this updated review is described in Table 5. All of the 27 studies that reported including public participants provided some detail about their participation. Among Delphi studies that reported participation of both clinical experts and the public, levels of participation for public participants ranged from 6% [8] to 69% [9].
Table 5. Nature of patient participation where detail is reported (n = 27).
| Methods used | Total number of participants | Number of public participants | % Public participants when multiple stakeholder groups included | |
|---|---|---|---|---|
| Agiostratidou | Survey | Not reported | 8 | Unknown |
| Allin | Delphi | Round 1: 108 | Round 1: 61 | 57% |
| Round 2: 96 | Round 2: 51 | 53% | ||
| Round 3: 89 | Round 3: 46 | 52% | ||
| Consensus meeting | 17 | 6 | 35% | |
| Measurement meeting | 14 | 4 | 29% | |
| Ammendolia | Interviews* | 28 | 28 | |
| Avery | Interviews | 38 | 31 | 82% |
| Delphi | Round 1: 188 | Round 1: 116 | 62% | |
| Round 2: 161 | Round 2: 94 | 58% | ||
| Consensus meeting* | 20 | 20 | ||
| Benstoem | Delphi | Round 1: 86 | Round 1: NR | 16% |
| Round 2: 46 | Round 2: NR | 16% | ||
| Round 3: 39 | Round 3: NR | 16% | ||
| Byrne | Delphi | Round 1: 132 | Round 1: 34 | 26% |
| Round 2: 81 | Round 2: 17 | 21% | ||
| Consensus meeting | 12 | 3 | 25% | |
| de Graaf | Interviews* | 26 | 26 | |
| Delphi^ | Round 1: 126 | Round 1: 0 | ||
| Round 2: 97 | Round 2: 0 | |||
| Egan | Delphi | Round 1: 151 | Round 1: 20 | 13% |
| Round 2: 120 | Round 2: NR | Unknown | ||
| Round 3: 101 | Round 3: NR | Unknown | ||
| Consensus meeting | 14 | 2 | 14% | |
| Grieve | Workshops (x4) | W1: 27 | W1: 5 | 19% |
| W2: 15 | W2: 1 | 7% | ||
| W3: 20 | W3: 5 | 25% | ||
| W4: 18 | W4: 3 | 17% | ||
| Klokkerud | Delphi consensus meeting | 46 | 6 | 13% |
| Layton | Delphi | Round 1: 500 | Round 1: 307 | 61% |
| Round 2: 164 | Round 2: 52 | 32% | ||
| Round 3: 116 | Round 3: 34 | 29% | ||
| MacLennan | Interviews* | 15 | 15 | |
| Delphi | Round 1: 174 | Round 1: 118 | 68% | |
| Round 2: 158 | Round 2: 109 | 69% | ||
| Round 3: 152 | Round 3: 105 | 69% | ||
| Meeting | 21 | 8 | 38% | |
| Marrie | Survey | 296 | 20 | 7% |
| Workshop | Not reported | Not reported | unknown | |
| McNamara | Delphi | 17 | 1 | 6% |
| Millar | Focus groups | 72 | 14 | 19% |
| Interviews^ | 13 | 0 | ||
| Delphi | Round 1: 19 | Round 1: 2 | 11% | |
| Round 2: 18 | Round 2: NR | |||
| Nabbout | Interviews | 11 | 7 | 64% |
| Delphi^ | Round 1: 8 | Round 1: 0 | ||
| Round 2: 7 | Round 2: 0 | |||
| Nikiphorou | Workshops and teleconference | 26 | 2 | 8% |
| Obbarius | Delphi | 24 | 2 | 8% |
| Interviews | not reported | not reported | ||
| Ong | Focus groups* | 8 | 8 | |
| Survey* | 1225 | 1225 | ||
| Delphi | not reported | not reported | ||
| Page | Delphi | Round 1: 91 | Round 1: 41 | 45% |
| Round 2: 96 | Round 2: 41 | 43% | ||
| Sreih | Delphi^ | 99 | 0 | |
| Interviews/focus groups* | 31 | 31 | ||
| Steutel | Delphi | Round 1: 188 | Round 1: 55 | |
| Round 2: 97 | Round 2: 43 | |||
| Meeting^ | not reported | 0 | ||
| Tong | NGT/focus groups* | 57 | 57 | |
| Delphi | Round 1: 1018 | Round 1: 461 | 45% | |
| Round 2: 844 | Round 2: 387 | 46% | ||
| Round 3: 779 | Round 3: 360 | 46% | ||
| Workshops | 124 | 27 | 22% | |
| Turnbull | Survey | 279 | 158 | 57% |
| Delphi | 77 | 19 | 25% | |
| Interviews* | 48 | 48 | ||
| Survey^ | 178 | 0 | ||
| Meeting^ | 129 | 0 | ||
| van der Poel | Delphi^ | Round 1: 30 | Round 1: 30 | |
| Round 2: 30 | Round 2: 30 | |||
| Consensus meeting | 16 | 1 | 6% | |
| Wallace | NGT* | 68 | 68 | |
| Delphi^ | Round 1: 318 | Round 1: 0 | ||
| Round 2: 153 | Round 2: 0 | |||
| Round 3: 137 | Round 3: 0 | |||
| Delphi^^ | Round 1: 72 | Round 1: 0 | ||
| Round 2: 63 | Round 2: 0 | |||
| Round 3: 61 | Round 3: 0 | |||
| Webster | Focus groups* | 18 | 18 | |
| Consensus conference^ | 27 | 0 | ||
| Consensus conference^ | 27 | 0 |
*patient only.
^clinician only.
^^researcher only.
Table 6 lists the geographical locations, by continent, of the participants involved in developing the COS in this update and in the combined reviews, as reported by the included studies. In the current review, participant locations were provided in 41 of the 48 studies, however for the remaining seven studies, location information was only provided for the authors. The majority of COS were developed with the inclusion of participants located in Europe (n = 38; 93%) and North America (n = 28; 68%). However, input from participants located in Australasia (n = 17; 41%), Asia (n = 18; 44%), South America (n = 13; 32%) and Africa (n = 7; 17%) has increased since the previous reviews. The number of countries that have been involved in the development of a COS ranges from 1 to 76 (a median of 6).
Table 6. Geographical locations of participants included in the development of each COS (n = 256).
| Locations | Original review n (%) |
Update review 1 n (%) |
Update review 2 n (%) |
Update review 3 n (%) |
Update review 4 n (%) |
Combined* N (%) |
|---|---|---|---|---|---|---|
| North America | 134 (82) | 17 (68) | 9 (64) | 6 (55) | 28 (68) | 196 (77) |
| Europe | 125 (76) | 19 (76) | 13 (93) | 10 (91) | 38 (93) | 206 (81) |
| Australasia | 42 (26) | 4 (16) | 5 (36) | 3 (27) | 17 (41) | 74 (29) |
| Asia | 34 (21) | 3 (12) | 6 (43) | 1 (9) | 18 (44) | 63 (25) |
| South America | 16 (10) | 3 (12) | 2 (14) | 1 (9) | 13 (32) | 37 (15) |
| Africa | 10 (6) | 1 (4) | 2 (14) | 1 (9) | 7 (17) | 22 (9) |
| Total | 164 (84) | 25 (89) | 14 (70) | 11 (73) | 41 (85) | 256 (83) |
| No details provided | 32 (16) | 3 (11) | 6 (30) | 4 (27) | 7 (15) | 51 (17) |
| Median and range of number of countries | 6, 1–76 | 2, 1–33 | 6, 1–28 | 2, 1–18 | 6, 1–37 | 6, 1–76 |
*Additional information provided by updated papers linked to previously published COS are reflected in the combined column.
Discussion
This systematic review update has identified 48 new COS articles that describe 56 new COS. This is a marked increase in the number of new COS identified in the previous three updates, which included 28, 20 and 15 new COS publications (describing the development of 29, 20 and 15 COS) respectively [2, 4, 5]. Similarly, there has been a continued increase in COMET website and database visits, new visitors, number of database searches and visits from around the world in 2017 (Source of data usage: Google Analytics), with 24,884 visits; 15,546 new visitors; 5142 database searches, and visitors from 139 countries. The sustained growth in use suggests that the COMET website and database are continuing to raise awareness about the existence of, and need for, COS; which could explain this increase in the number of COS published in 2017 and subsequently included in this review. Their inclusion in the database brings the overall total of COS publications to 307, relating to 366 COS.
This review has identified an improvement in the reporting of COS scope (population and intervention characteristics), stakeholder participants and methods used. Core Outcome Set-STAndards for Reporting (COS-STAR) were published in 2016 [10]. This is the first update since the publication of the COS-STAR guidelines, and it is encouraging to see that ten of the 48 studies (21%) included in this update referenced these reporting guidelines. This provides a baseline with which to compare future updates, where we hope to see a continued increase in the number of studies reporting their work using these guidelines.
The previous update highlighted an increase in the use of mixed methods, including the Delphi method, for COS development [5]. A recent survey of ongoing COS developers indicated that mixed methods, including Delphi, were commonly used in ongoing COS development, and there is an increased inclusion of public participants in ongoing studies [11]. As such, the percentage of new studies including public participation is expected to rise in future updates of this systematic review as ongoing studies are published. Participation of non-clinical research experts has increased, and as one of the criteria included in the recently published minimum standards for COS development [12], we would expect that this should continue to increase in new COS studies included in any further updates to this review. The Core Outcome Set-STAndards for Development (COS-STAD) recommendations will help users to assess whether a particular COS has been developed using a reasonable approach.
The involvement of stakeholders from a range of countries, but in particular lower and middle income countries, has previously been identified as a gap in COS development. [5] This update has shown an increase in participation from Australasia, Asia, South America and Africa. A recent survey to identify the top priorities for trials methodology research in LMICs found that choosing appropriate outcomes to measure was one of the two most important topics [6]. The involvement of LMIC countries in COS development and uptake is therefore a key area of importance and efforts should be made to improve this.
In conclusion, we have completed the fourth update to a systematic review of COS, identifying COS published and indexed in 2017. COS methodology, and consensus development methodology more generally, have developed over recent years. This is reflected in the improvements seen throughout these updates, not only in the reporting of important COS details (such as scope), but in the methods chosen for development and the stakeholders who participate in the process. With the recent development and publication of guidelines and minimum standards for COS development [1, 12], it is hoped that improvements in these areas will continue.
Supporting information
PRISMA checklist for content of a systematic review.
(DOC)
(DOCX)
(DOCX)
Acknowledgments
We would like to acknowledge Jane Blazeby (Bristol University) and Mike Clarke (Queen’s University Belfast) for their involvement in the conceptualisation and methodology in the original systematic review on which this update is based.
We would like to thank Yalda Ashraf Kharaz (University of Liverpool) for help with document translation of one paper. We also acknowledge Jamie Kirkham (University of Liverpool) for checking for correct exclusion in this review update.
Data Availability
All relevant data are within the manuscript and its Supporting Information files.
Funding Statement
This work was supported by the National Institute for Health Research Senior Investigator Award (NF-SI_0513-10025) and the Medical Research Council North West Hub for Trials Methodology Research grant (MR/K025635/1). Karen Matvienko-Sikar is funded by a Health Research Board Interdisciplinary Capacity Enhancement Award (2015-1026). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
References
- 1.Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(Suppl 3):280 Epub 2017/07/07. 10.1186/s13063-017-1978-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR. Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and User Survey. PLoS ONE. 2016;11(1):e0146444 Epub 2016/01/20. 10.1371/journal.pone.0146444 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, et al. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS ONE. 2014;9(6):e99111 Epub 2014/06/17. 10.1371/journal.pone.0099111 ; PubMed Central PMCID: PMC4059640. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Gorst SL, Gargon E, Clarke M, Smith V, Williamson PR. Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and Identification of Gaps. PLoS One. 2016;11(12):e0168403 Epub 2016/12/16. 10.1371/journal.pone.0168403 ; PubMed Central PMCID: PMC5156438 to support COMET and related work. EG is a member of the COMET Management Group and is the COMET Project Co-ordinator. This does not alter the authors' adherence to PLOS ONE policies on sharing data and materials. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Davis K, Gorst SL, Harman N, Smith V, Gargon E, Altman DG, et al. Choosing important health outcomes for comparative effectiveness research: An updated systematic review and involvement of low and middle income countries. PLoS One. 2018;13(2):e0190695 Epub 2018/02/14. 10.1371/journal.pone.0190695 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Rosala-Hallas A, Bhangu A, Blazeby J, Bowman L, Clarke M, Lang T, et al. Global health trials methodological research agenda: results from a priority setting exercise. Trials. 2018;19(1):48 Epub 2018/02/06. 10.1186/s13063-018-2440-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097 Epub 2009/07/22. 10.1371/journal.pmed.1000097 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.McNamara RL, Spatz ES, Kelley TA, Stowell CJ, Beltrame J, Heidenreich P, et al. Standardized Outcome Measurement for Patients With Coronary Artery Disease: Consensus From the International Consortium for Health Outcomes Measurement (ICHOM). Journal of the American Heart Association. 2015;4(5). Epub 2015/05/21. 10.1161/jaha.115.001767 ; PubMed Central PMCID: PMC4599409. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.MacLennan S, Williamson PR, Bekema H, Campbell M, Ramsay C, N'Dow J, et al. A core outcome set for localised prostate cancer effectiveness trials. BJU international. 2017;120(5b):E64–e79. Epub 2017/03/28. 10.1111/bju.13854 . [DOI] [PubMed] [Google Scholar]
- 10.Kirkham JJ, Gorst S, Altman DG, Blazeby JM, Clarke M, Devane D, et al. Core Outcome Set-STAndards for Reporting: The COS-STAR Statement. PLoS Med. 2016;13(10):e1002148 Epub 2016/10/19. 10.1371/journal.pmed.1002148 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Biggane AM, Brading L, Ravaud P, Young B, Williamson PR. Survey indicated that core outcome set development is increasingly including patients, being conducted internationally and using Delphi surveys. Trials. 2018;19(1):113 Epub 2018/02/20. 10.1186/s13063-018-2493-y ; PubMed Central PMCID: PMC5816387. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kirkham JJ, Davis K, Altman DG, Blazeby JM, Clarke M, Tunis S, et al. Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med. 2017;14(11):e1002447 Epub 2017/11/18. 10.1371/journal.pmed.1002447 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
PRISMA checklist for content of a systematic review.
(DOC)
(DOCX)
(DOCX)
Data Availability Statement
All relevant data are within the manuscript and its Supporting Information files.