Abstract
Background
Core outcome sets (COS) comprise a minimum set of outcomes that should be measured and reported in all trials for a specific health condition. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative maintains an up to date, publicly accessible online database of published and ongoing COS. An annual systematic review update is an important part of this process.
Methods
This review employed the same, multifaceted approach that was used in the original review and the previous two updates. This approach has identified studies that sought to determine which outcomes/domains to measure in clinical trials of a specific condition. This update includes an analysis of the inclusion of participants from low and middle income countries (LMICs) as identified by the OECD, in these COS.
Results
Eighteen publications, relating to 15 new studies describing the development of 15 COS, were eligible for inclusion in the review. Results show an increase in the use of mixed methods, including Delphi surveys. Clinical experts remain the most common stakeholder group involved. Overall, only 16% of the 259 COS studies published up to the end of 2016 have included participants from LMICs.
Conclusion
This review highlights opportunities for greater public participation in COS development and the involvement of stakeholders from a wider range of geographical settings, in particular LMICs.
Introduction
Measuring appropriate outcomes in clinical trials enables the benefits and harms of specific treatments to be compared between trials and allows decision makers, such as patients and clinicians, to be best informed in their choice. This is known as comparative effectiveness research (CER) [1]. Heterogeneity of outcome reporting in trials is, however, common, even in trials exploring the effects of the same intervention on a specific disease [2]. The lack of comparability that results from this causes waste in research [3,4]. This waste could be avoided with the development and systematic use of core outcome sets (COS) in clinical trials research, as demonstrated by the uptake in trials of a COS developed in rheumatoid arthritis [5]. A COS represents an agreed minimum set of outcomes that should be measured and reported in all trials for a specific health condition [2]. An element of the scope of the COS, to be defined, is whether the COS is to be applicable to all interventions or exclusively to specific intervention types. The use of COS would ensure all trials produce evidence that can be combined appropriately with the results of other trials, while not restricting the researchers’ ability to explore other, additional outcomes [2].
The purpose of the COMET (Core Outcome Measures in Effectiveness Trials) Initiative is to encourage the development, promotion and application of COS across all health areas. It does so by maintaining an up-to-date, online, publicly-accessible database of ongoing and published COS studies. Three systematic reviews have comprehensively searched for COS; the original review, conducted in 2013 [6], and two updates, one conducted in 2015 [7] and the second conducted in 2016 [8].
The 2016 update showed that COS had been developed for 13 of the world’s 25 most prevalent health conditions, leaving 12 of the 25 as being in need of COS development [8]. Eighty-two percent of the COS applicable to the world’s most prevalent health conditions involved participants from Europe and North America only. International participation in COS was highlighted as a key research gap. Representation of stakeholders from Africa and South America was particularly low in comparison to other continents, at only one and three studies (respectively). Further research is needed to establish how COS can be developed to have the greatest global relevance and applicability to the global burden of disease.
Aims
The aims of the current study were to (i) update the systematic review [6–8] in order to identify any further studies where a COS has been developed; (ii) to describe the methodological approaches taken in these studies, (iii) to identify countries in which COS development participants are located and (iv) to highlight areas for future COS development and improvement.
Methods
Systematic review update
The methods used in this updated review followed the same approach used in the original review and in the previous two updates [6–8].
Study selection
Inclusion and exclusion criteria: As described in detail previously [6], studies were eligible for inclusion if they developed or applied methodology for determining which outcome domains or outcomes should be measured, or are important to measure, in clinical trials or other forms of health research. As noted in the 2016 update [8], by using the term `outcome' we are referring to something that occurs as a result of the specific health condition (e.g. diarrhoea) and by `outcome domain' we are referring to the grouping of individual outcomes (e.g. bowel function, which would include diarrhoea).
Types of participants and interventions: As described previously [6], studies were categorised as eligible if they reported the development of a COS, regardless of any restrictions by age, health condition or setting, which could be used to assess the effect of interventions for that condition.
Identification of relevant studies
In March 2017, we searched MEDLINE via Ovid and SCOPUS (including EMBASE) without language restrictions. The search identified studies that had been published or indexed between the previous systematic review update [8] in January 2016 and the end of December 2016. The multifaceted search strategy, developed for the original review using a combination of text words and index terms and combining three concepts of search terms that cover ‘randomised trial’, ‘systematic review’, ‘methodology’ and ‘outcomes’ [9] was used in the current update, with adaptations appropriate for each database (S1 Table). The search was deferred until March 2017 due to indexing issues that had been identified with the previous reviews [8], namely that studies published at the end of a particular year of interest had not been indexed in the database until the beginning of the following year and hence were not captured by the database search. We searched in March for this update to try to overcome this limitation. In addition to this database searching, we also completed hand searching activities as previously described [8]. We identified any studies that had been submitted directly to the COMET database. We also examined references cited in eligible studies and in ineligible studies that referred to or used a COS.
Selecting studies for inclusion in the review
Records from each database were combined and duplicates were removed. Titles and abstracts were read to assess eligibility of studies for inclusion in the review (stage 1). Full texts of potentially relevant articles were obtained to assess for inclusion (stage 2). Titles and abstracts were divided between reviewers (KD, NH, VS and PRW) who independently checked the title and abstract of each citation assigned to them and classified them as include, exclude or unsure. Where there was uncertainty in the classification of a record, a second reviewer was consulted. If agreement could not be achieved, the citation was referred to a fifth reviewer (SG). Full papers were divided between three reviewers (KD, NH and VS) for full-paper review. Reasons for exclusion at this stage were documented for articles judged to be ineligible.
Checking for agreement between the reviewers
During each stage of the review process, agreement between reviewers was assessed. Prior to independently assessing records, the four reviewers (KD, NH, VS and PRW) independently checked batches of abstracts and full papers to confirm consistency. This was repeated in batches of 10 until complete agreement was reached in three consecutive batches.
Checking for correct exclusion
Of the records that had been excluded on the basis of the title and abstract, full text papers were obtained for a 1% sample and a fifth reviewer (SG) assessed correct exclusion. If any studies were identified as being incorrectly excluded, further checking was performed within the other excluded records. Of the records that had been excluded after reading their full text papers, 5% were assessed for correct exclusion at that stage. In this review, no studies were excluded incorrectly.
Data extraction
As described in detail previously [6], data were extracted in relation to the study aim(s), health area, target population, interventions covered, methods of COS development and stakeholder groups involved. Data relating to the geographical locations of participants included in the development of COS were extracted for studies found in the current update, as was done in the previous reviews.
Data analysis and results
As previously [6], we used a narrative analysis and the results are presented descriptively. The results were analysed for the scope of their aims, intended use, population characteristics and intervention characteristics. The methods used, and stakeholders involved in selecting outcomes were also analysed. This update also included an analysis of the inclusion of participants from low and middle income countries (LMICs), as identified by the OECD, in these COS.
Results
Description of studies
Following the removal of duplicates, 4406 citations were identified in the database search. A total of 3887 records were excluded during the title and abstract stage, and a further 503 were excluded following the assessment of full text papers (Fig 1). Table 1 provides a summary of the reasons for exclusion of papers at abstract and full text stage. Sixteen citations relating to 13 new studies met the inclusion criteria. In addition to the database search, two additional citations were identified as being eligible for inclusion in the review. In total, 18 reports relating to 15 new studies describing the development of 15 COS were added to the review during this update (S2 Table).
Fig 1. Identification of studies.
Table 1. Reasons for exclusion at full text stage.
| Exclusion Categories of Full Text Stage | Number of Papers |
|---|---|
| Studies relating to how, rather than which, outcomes should be measured | 21 |
| Studies reporting the design/ rationale of single trial | 1 |
| Studies reporting the use of a COS | 2 |
| Systematic reviews of clinical trials | 13 |
| Systematic review of prognostic studies | 2 |
| Review/overview/discussion only, no outcome recommendations | 37 |
| Core outcomes/ outcome recommendations not made | 63 |
| Quality indicators–structure and/or process of care only | 10 |
| One outcome/ domain only | 3 |
| Instrument development | 4 |
| Recommendations by single author only | 1 |
| ICF Core set | 0 |
| Describes features of registry | 9 |
| Preclinical/ Early phase only (0, I, II) | 5 |
| Quantitative description | 1 |
| Value attributed to outcomes | 0 |
| Irrelevant | 271 |
| Assessed in previous review | 0 |
| Systematic review of outcomes used in studies | 12 |
| HRQL | 3 |
| Recommendations for clinical management in practice not research | 31 |
| Studies that elicit stakeholder group opinion regarding which outcome domains or outcomes are important | 10 |
| Ongoing studies | 4 |
Three of the reports included in this updated are linked to COS studies that were reported in previous reviews [6,7]. One report is linked to a previously published preliminary COS, which has since been modified and endorsed [10]. Two further reports are linked to previously published COS, which have been revised to meet the current standards of outcome measure development [11,12]. All updates and revisions reported in the linked papers are reflected in the tables below.
Included studies
Year of publication
Our analysis of the year of first publication of each COS study included in the previous reviews has been updated to include the 15 new studies identified in this updated review (Fig 2). Of the 15 studies discussed here, 13 studies were published in 2015 or 2016, and two studies were published in 2014. Two studies were identified through reference checking of a COS study. One of these studies had been excluded from a previous review at the title and abstract screening stage because of the lack of COS information in the abstract [13]. The other study was identified through reference checking of related COS studies and had not come up in any of the previous review searches [14]. As referred to by the study authors, the scope of this COS, specifically the health condition, overlaps with earlier COS.
Fig 2. Year of first publication of each COS study (n = 259).
Scope of core outcome sets
The scope of published COS studies is summarised in Table 2 and includes 259 COS studies (describing the development of 309 COS) that were included in the three previous systematic reviews and the 15 new COS studies that have been added by this updated review. Please see S2 Table in supplementary material for the scope of the disease categories and names relevant to each of the 15 new COS. During this update, five studies that were judged to be COS in previous reviews were reviewed and excluded [15–19]. One was retrospectively deemed to refer to how to measure outcomes rather than what to measure [18] and four others were deemed to be composite end points rather than core outcome sets [15–17,19].
Table 2. The scope of included studies (n = 259).
| Update review 3 n (%) | Combined n (%) |
|
|---|---|---|
| Study aims | ||
| Specifically considered outcome selection and measurement | 10 (60) | 142 (54) |
| Considered outcomes while addressing wider clinical trial design issues | 5 (40) | 117 (46) |
| Intended use of recommendations | ||
| Clinical research | 9 (60) | 226 (87) |
| Clinical research and practice | 6 (40) | 33 (13) |
| Population characteristics | ||
| Adults | 6 (40) | 36 (14) |
| Children | 0 (0) | 30 (12) |
| Adults and children | 8 (53) | 25 (10) |
| Older adults | 0 (0) | 3 (1) |
| Adolescents and young adults | 1 (7) | 1 (<1) |
| Not specified | 0 (0) | 164 (63) |
| Intervention characteristics | ||
| All intervention types | 8 (53) | 39 (15) |
| Drug treatments | 0 (0) | 42 (16) |
| Surgery | 4 (27) | 25 (10) |
| Vaccine | 0 (0) | 2 (<1) |
| Rehabilitation | 1 (7) | 3 (1) |
| Exercise | 0 (0) | 3 (1) |
| Procedure | 0 (0) | 6 (2) |
| Device | 1 (7) | 4 (1) |
| Other | 1 (7) | 17 (7) |
| Not specified | 0 (0) | 118 (46) |
Methods used to select outcomes
The methods used to develop the 15 new COS are presented in Table 3 alongside the methods used in the combined reviews [6–8]. The results show an increase in the use of mixed methods, including Delphi surveys.
Table 3. The methods used to develop COS (n = 259).
| Main methods | Update review 3 n (%) | Combined n (%) |
|---|---|---|
| Semi-structured group discussion only | 59 (23) | |
| Unstructured group discussion only | 18 (7) | |
| Consensus development conference only | 13 (5) | |
| Literature/systematic review only | 1 (7) | 19 (7) |
| Delphi only | 10 (4) | |
| Survey only | 3 (1) | |
| NGT only | 1 (<1) | |
| Mixed methods (see descriptions below)1 | 12 (80) | 116 (45) |
| Delphi + another method(s) | 9 (60) | 48 (19) |
| Semi-structured group discussion + another method(s) | 2 (13) | 42 (16) |
| Consensus development conference + another method(s) | 7 (3) | |
| Literature/systematic review + another method(s) | 1 (7) | 14 (5) |
| NGT + another method(s) | 4 (2) | |
| Focus group + another method(s) | 1 (<1) | |
| No methods described | 2 (13) | 20 (8) |
1 Mixed methods studies are reported cumulatively. For example if they have been included in one subcategory they will be excluded from subsequent categories even if they apply.
Stakeholders involved in selecting outcomes
Table 4 lists the stakeholders that were involved in selecting outcomes for inclusion in the COS identified in this update and the combined reviews. Regarding the 259 published COS studies, 225 have provided details about the stakeholders who participated in the development process. Of these 225 studies, clinical experts have been involved in selecting outcomes for inclusion in 222 (99%) studies; this contrasts with public representatives, who have been included in only 62 (28%) studies.
Table 4. Participant groups involved in selecting outcomes for inclusion in COS (n = 225).
| Participants category | Sub-category (not mutually exclusive) | Frequency of participants | |
|---|---|---|---|
| Update review 3 n (%)1 |
Combined n (%)2 |
||
| Clinical experts | 14 (93) | 222 (99) | |
| Clinical experts | 14 | 131 (58) | |
| Clinical research expertise | 2 | 87 (39) | |
| Clinical trialists/Members of a clinical trial network | 11 (5) | ||
| Others with assumptions | 54 (24) | ||
| Public representatives | 8 (53) | 62 (28) | |
| Patients | 8 | 45 (20) | |
| Carers | 3 | 14 (6) | |
| Patient support group representatives | 15 (7) | ||
| Service users | 1 | 3 (1) | |
| Non-clinical research experts | 2 (13) | 73 (32) | |
| Researchers | 2 | 37 (16) | |
| Statisticians | 26 (12) | ||
| Epidemiologists | 14 (6) | ||
| Academic research representatives | 4 (2) | ||
| Methodologists | 11(5) | ||
| Economists | 4 (2) | ||
| Authorities | 0 | 47 (21) | |
| Regulatory agency representatives | 37 (16) | ||
| Governmental agencies | 13 (6) | ||
| Policy makers | 5 (2) | ||
| Charities | 1 (<1) | ||
| Industry representatives | 0 | 38 (17) | |
| Pharmaceutical industry representatives | 34 (15) | ||
| Device manufacturers | 3 (1) | ||
| Biotechnology company representatives | 1 (<1) | ||
| Others | 1 (7) | 76 (34) | |
| Ethicists | 1 (<1) | ||
| Journal editors | 3 (1) | ||
| Funding bodies | 1 (<1) | ||
| Yoga therapists/instructors | 1 (<1) | ||
| Members of health care transition research consortium | 1 | 1 (<1) | |
| Others (besides known participants) | 15 (7) | ||
| Others with assumptions | 54 (24) | ||
| No details given | 0 (0) | 34 (15) | |
1Studies providing details about participants groups involved in selecting outcomes (n) = 15
2Studies providing details about participants groups involved in selecting outcomes (n) = 225
Public representatives include patients, carers, health and social care service users and people from organisations who represent these groups. The degree of public participation within the development of the COS studies included in this updated review is described in Table 5. Seven of the eight studies that reported including public participants, provided details about their participation. Among studies that report public participation, levels of participation range from 15% [20] to 66% [21]. Where patients are involved, they tend to make up a greater percentage of the participants than in COS reported in previous reviews.
Table 5. Public participation detail where reported (n = 7).
| Methods used | Total number of participants | Number of public participants | % Public participants | |
|---|---|---|---|---|
| 1 | Delphi (clinician only) | Round 1: 75 | Round 1: 0 | |
| Round 2: 48 | Round 2: 0 | |||
| Delphi (mixed)1 | Round 3: 61 | Round 3: 24 | 39% | |
| Meeting (mixed) | 16 | 2 | 15% | |
| 2 | Delphi (mixed) | Round 1: 258 | Round 1: 90 | 35% |
| Round 2: 200 | Round 2: 80 | 40% | ||
| Round 3: 173 | Round 3: 71 | 41% | ||
| Meeting (clinician only) | 33 | 0 | ||
| Meeting (patient only) | 9 | 9 | ||
| 3 | Meeting (clinician only) | 16 | 0 | |
| Survey (clinician only) | 16 | 0 | ||
| Survey (patient only) | 0 | 221 | ||
| 4 | Delphi (mixed) | Round 1: 228 | Round 2: 150 | 66% |
| Round 2: 208 | Round 2: 135 | 65% | ||
| 5 | Delphi (mixed) | Round 1: 115 | Round 1: 55 | 48% |
| Round 2: 101 | Round 2: 46 | 46% | ||
| Round 3: 86 | Round 3: 35 | 41% | ||
| 6 | Delphi (mixed) | Round 1: 195 | Round 1: 97 | 50% |
| Round 2: 165 | Round 2: 87 | 53% | ||
| Meeting (clinician only)2 | 61 | 0 | ||
| 35 | 0 | |||
| Meeting (patient only) | 14 | 14 | ||
| 7* | Survey (patient only) | 615 | 615 |
1Patients brought in at round 3.
2 A second clinician meeting was held as consensus was not reached within the allotted time during the first clinician meeting
* Patient core set
Countries involved in the development of COS
Table 6 lists the geographical locations, by continent, of the participants involved in developing the COS in this update and in the combined reviews, as reported by the included studies.
Table 6. Geographical locations of participants included in the development of each COS (n = 214).
| Continents | Update review 3 n (%) |
Combined n (%) |
|---|---|---|
| North America | 6 (55) | 167 (78) |
| Europe | 10 (91) | 167 (78) |
| Australasia | 3 (27) | 56 (26) |
| Asia | 1 (9) | 45 (21) |
| South America | 1 (9) | 24 (11) |
| Africa | 1 (9) | 15 (7) |
| No details provided | 4 (27) | 45 (17) |
Fig 3 displays a choropleth map of the world, which uses differences in shading to indicate the number of COS studies that have been developed with the inclusion of participants from each country. Of the 259 COS studies published by the end of 2016, the countries which have been most involved in the COS development process are the United States of America (USA) (n = 161) and the United Kingdom (UK) (n = 124). There are 93 countries which are displayed on the map in the lightest shading; these represent the locations that have not been included in the development of any COS.
Fig 3. Number of COS studies that have included participants from each individual country throughout the world.
Of the 259 published COS studies, 41 (16%) have included participants from low and middle income countries (LMICs). These LMICs were identified according to the Organization for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) list of Official Development Assistance (ODA) recipients [22]. Table 7 presents a breakdown of the LMICs that have been involved in the development of COS.
Table 7. OECD DAC list of ODA recipient countries involved in the COS development process.
| DAC1 list classifications | Number of COS involving participants | Countries |
|---|---|---|
| Least Developed Countries | 4 | Afghanistan, Burkina Faso, Ethiopia, Gambia, Liberia, Malawi, Mali, Mozambique, Nepal, Senegal, Tanzania |
| Other Low Income Countries | 3 | Kenya, Zimbabwe |
| Lower Middle Income Countries | 17 | Cameroon, Egypt, Georgia, Ghana, Guatemala, India, Indonesia, Morocco, Nigeria, Pakistan, Philippines, Sri Lanka, Ukraine |
| Upper Middle Income Countries | 39 | Albania, Algeria, Argentina, Brazil, Chile, China, Colombia, Costa Rica, Cuba, Ecuador, Iran, Lebanon, Malaysia, Mexico, Panama, Peru, Serbia, South Africa, Thailand, Tunisia, Turkey, Uruguay, Venezuela |
1 Development Assistance Committee
Discussion
This update to the previous three COMET Initiative systematic reviews has identified 18 publications relating to 15 new COS, suggesting that COS continue to be developed and published.
A range of resources are available on the COMET website to facilitate COS development. These include a core resource pack, plain language summaries and information about how to optimise the involvement of parents, young people and patient organisations in COS studies. This work has now been brought together in the COMET Handbook [23] to provide guidance on the development, application and implementation of COS.
Of the 259 COS studies published up to the end of 2016, the USA and the UK continue to be the most frequently represented countries by stakeholders involved. Use of the COMET website is continuing to increase, with the number of visitor locations reaching 175 countries in 2017 [24], however there are 96 countries that have not been included in the development of any COS. COS developed with globally rather than nationally located participants might improve the global applicability of clinical trials. To be comprehensive, a COS should be applicable in all relevant settings, including across different countries where appropriate. COS are more likely to be applicable within those countries that the participants in the development process are from. Therefore, if a COS is intended to have international applicability, this would have implications for how it is developed, who is involved, and the resources required for the development process. In particular, the developers should involve participants from countries where the prevalence or burden of the disease is high, but, with this in mind, it is worth noting that only 16% of COS developed to date have included participants from LMICs. Furthermore, COS developed with participants from multiple countries might improve the global applicability of the COS and, consequently, the global relevance and impact of the clinical trials that these are used in. A survey has recently been undertaken to identify the top priorities for trials methodological research in LMICs to inform further research and ultimately to improve clinical trials in these regions. The priority most commonly graded as critically important, amongst 400 participants from over 80 countries, was choosing appropriate outcomes to measure [25].
Implications
The 15 new COS studies have been added to the COMET database, ensuring that it will remain up to date and continue to assist trialists, researchers, clinicians and others to design clinical trials, guidelines and systematic reviews. It will also remain a key source of information for those looking to develop a COS. Although reporting has improved, use of the COS-STAR guideline [26] will help further.
The updated review reported here highlights a continuing need to involve participants from a greater range of geographical locations in COS development, particularly from LMICs, in order to increase the global relevance of COS.
Limitations
Gathering accurate data on the geographic location of participants is dependent on the reporting of this information, which was provided for over 70% of the COS studies in this review update. It is possible that there could be more COS involving participants from LMICs than reported here, although, there is no obvious reason why those reporting geographic information may differ with respect to LMIC involvement.
The abstracts reviewed in this study were reviewed by a single author only, in contrast to previous reviews where two reviewers reviewed each abstract. It is conceivable that some COS may have been missed at this stage of screening, however checks have been put in place to minimise incorrect exclusion, including batch checks and additional reviewers checking samples of excluded studies. The impact of this lack of identification on the usefulness of the COMET database is minimised, as the annual systematic review update is only one of a number of methods used to populate the COMET database, which also include database alerts and direct contact with COS developers throughout the year.
Conclusions
In conclusion, this study has demonstrated the continuing development of new COS. The results show an increase in the use of mixed methodologies for COS development but suggest a need to push for greater public participation and the involvement of stakeholders from a broader range of countries, in particular LMICs. These efforts will increase the applicability of COS to global health and tackling the global burden of disease.
Supporting information
(DOC)
(DOCX)
(DOCX)
Data Availability
All relevant data are within the paper and its Supporting Information files.
Funding Statement
This work was funded by the NIHR (National Institute for Health Research, https://www.nihr.ac.uk/), grant number NF-SI_0513-10025. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
References
- 1.Sox HC, Greenfield S. Comparative effectiveness research: a report from the Institute of Medicine. Ann Intern Med. 2009;151(3):203–205. [DOI] [PubMed] [Google Scholar]
- 2.Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132 doi: 10.1186/1745-6215-13-132 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Chalmers I, Glasziou P. Avoidable waster in the production and reporting of research evidence. Lancet. 2009;374(9683):86–89. doi: 10.1016/S0140-6736(09)60329-9 [DOI] [PubMed] [Google Scholar]
- 4.Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. doi: 10.1016/S0140-6736(13)62329-6 [DOI] [PubMed] [Google Scholar]
- 5.Kirkham JJ, Boers M, Tugwell P, Clarke M, Williamson PR. Outcome measures in rheumatoid arthritis randomised trials over the last 50 years. Trials. 2013;14:324 doi: 10.1186/1745-6215-14-324 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, et al. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS ONE. 2014;9(6):e99111 doi: 10.1371/journal.pone.0099111 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR. Choosing important health outcomes for comparative effectiveness research: an updated review and user survey. PLoS ONE. 2016;11(1):e0146444 doi: 10.1371/journal.pone.0146444 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Gorst SL, Garon E, Clarke M, Smith V, Williamson PR. Choosing important health outcomes for comparative effectiveness research: an updated review and identification of gaps. PLoS ONE. 2016;11(12):e0168403 doi: 10.1371/journal.pone.0168403 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Gargon E, Williamson PR, Clarke M. Collating the knowledge base for core outcome set development: developing and appraising the search strategy for a systematic review. BMC Medical Research Methodology. 2015;15:26 doi: 10.1186/s12874-015-0019-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Khanna D, Mittoo S, Aggarwal R, Proudman SM, Dalbeth N, Matteson EL, et al. Connective Tissue Disease-associated Interstitial Lung Diseases (CTD-ILD)—Report from OMERACT CTD-ILD Working Group. The Journal of rheumatology. 2015; 42(11):2168–71. doi: 10.3899/jrheum.141182 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Milman N, Boonen A, Tugwell P, Merkel PA; OMERACT Vasculitis Working Group. Clinicians’ perspective on key domains in ANCA-associated vasculitis: a Delphi exercise. Scandinavian Journal of Rheumatology. 2017;46(2):112–7. doi: 10.1080/03009742.2016.1188980 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Orbai AM, de Wit M, Mease P, Shea JA, Gossec L, Leung YY, et al. International patient and physician consensus on a psoriatic arthritis core outcome set for clinical trials. Annals of the Rheumatic Diseases. 2017;76(4):673–80. doi: 10.1136/annrheumdis-2016-210242 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Myatt L, Redman CW, Staff AC, Hansson S, Wilson ML, Laivuori H, et al. Strategy for Standardization of Preeclampsia Research. Hypertension. 2014;63(6):1293–301. doi: 10.1161/HYPERTENSIONAHA.113.02664 [DOI] [PubMed] [Google Scholar]
- 14.Deyo R.A., et al. , Report of the NIH Task Force on Research Standards for Chronic Low Back Pain. Journal of Pain. 2014;15(6):569–85. doi: 10.1016/j.jpain.2014.03.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Billings FT, Shaw AD. Clinical trial endpoints in acute kidney injury. Nephron. 2014;127(1–4):89–93. doi: 10.1159/000363725 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Coolsen MM, Clermonts SH, van Dam RM, Winkens B, Malagó M, Fusai GK, et al. Development of a composite endpoint for randomized controlled trials in pancreaticoduodenectomy. World J Surg. 2014;38(6):1468–1475. doi: 10.1007/s00268-013-2421-y [DOI] [PubMed] [Google Scholar]
- 17.Fong F, Rogozinska E, Allotey J, Kempley S, Shah DK, Thangaratinam S. Development of maternal and neonatal composite outcomes for trials evaluating management of late-onset pre-eclampsia. Hypertens Pregnancy. 2014;33(2):115–131. doi: 10.3109/10641955.2013.837176 [DOI] [PubMed] [Google Scholar]
- 18.Scher HI, Eisenberger M, D'Amico AV, Halabi S, Small EJ, Morris M, et al. Eligibility and outcomes reporting guidelines for clinical trials for patients in the state of a rising prostate-specific antigen: recommendations from the Prostate-Specific Antigen Working Group. Journal of Clinical Oncology. 2004;22(3):537–556. doi: 10.1200/JCO.2004.07.099 [DOI] [PubMed] [Google Scholar]
- 19.Smaïl-Faugeron V, Fron Chabouis H, Durieux P, Attal JP, Muller-Bolla M, Courson F. Development of a core set of outcomes for randomized controlled trials with multiple outcomes—example of pulp treatments of primary teeth for extensive decay in children. PLoS ONE. 2013;8(1):e51908 doi: 10.1371/journal.pone.0051908 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Al Wattar BH, Tamilselvan K, Khan R, Kelso A, Sinha A, Pirie AM, et al. Development of a core outcome set for epilepsy in pregnancy (E-CORE): a national multi-stakeholder modified Delphi consensus study. BJOG: An International Journal of Obstetrics and Gynaecology. 2017;124(4):661–667. [DOI] [PubMed] [Google Scholar]
- 21.Gerritsen A, Jacobs M, Henselmans I, van Hattum J, Efficace F, Creemers GJ, et al. Developing a core set of patient-reported outcomes in pancreatic cancer: A Delphi survey. European journal of cancer. 2016;57:68–77. doi: 10.1016/j.ejca.2016.01.001 [DOI] [PubMed] [Google Scholar]
- 22.Organisation for Economic Co-operation and Development. Development Assistance Committee (DAC) list of Official Development Assistance (ODA) recipients. Available from http://www.oecd.org/dac/stats/daclist.htm. Accessed July 17, 2017.
- 23.Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Google. Google Analytics. 2017. Available from: https://analytics.google.com. Accessed July 17, 2017.
- 25.Rosala-Hallas A, Bhangu A, Bowman L, Blazeby J, Clarke M, Lang T, et al. Global Health Trials Methodological Research Agenda: Results from a priority setting exercise. Trials. Under Review. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Kirkham JJ, Gorst S, Altman DG, Blazeby J, Clarke M, Devane D, et al. Core Outcome Set–STAndards for Reporting: The COS-STAR Statement. PLoS Med. 2016;13(10):e1002148 doi: 10.1371/journal.pmed.1002148 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
(DOC)
(DOCX)
(DOCX)
Data Availability Statement
All relevant data are within the paper and its Supporting Information files.