Abstract
Background
Synthesizing outcomes of underreported primary studies can pose a serious threat to the validity of outcomes and conclusions of systematic reviews. To address this problem, the Cochrane Collaboration recommends reviewers to contact authors of eligible primary studies to obtain additional information on poorly reported items. In this protocol, we present a cross-sectional study and a survey to assess (1) how reviewers of new Cochrane intervention reviews report on procedures and outcomes of contacting of authors of primary studies to obtain additional data, (2) how authors reply, and (3) the consequences of these additional data on the outcomes and quality scores in the review. All research questions and methods were pilot tested on 2 months of Cochrane reviews and were subsequently fine-tuned.
Methods for the cross-sectional study
Eligibility criteria are (1) all new (not-updates) Cochrane intervention reviews published in 2016, (2) reviews that included one or more primary studies, and (3) eligible interventions refer to contacting of authors of the eligible primary studies included in the review to obtain additional research data (e.g., information on unreported or missing data, individual patient data, research methods, and bias issues). Searching for eligible reviews and data extraction will be conducted by two authors independently. The cross-sectional study will primarily focus on how contacting of authors is conducted and reported, how contacted authors reply, and how reviewers report on obtained additional data and their consequences for the review.
Methods for the survey
The same eligible reviews for the cross-sectional study will also be eligible for the survey. Surveys will be sent to the contact addresses of these reviews according to a pre-defined protocol. We will use Google Forms as our survey platform. Surveyees are asked to answer eight questions. The survey will primarily focus on the consequences of contacting authors of eligible primary studies for the risk of bias and Grading of Recommendations, Assessment, Development and Evaluation scores and the primary and secondary outcomes of the review.
Discussion
The findings of this study could help improve methods of contacting authors and reporting of these procedures and their outcomes. Patients, clinicians, researchers, guideline developers, research sponsors, and the general public will all be beneficiaries.
Electronic supplementary material
The online version of this article (10.1186/s13643-017-0643-z) contains supplementary material, which is available to authorized users.
Keywords: Systematic review, Meta-analysis, Methodology, Author contact, Replying author, Risk of bias, Missing data, Reporting, Cochrane Collaboration, Email address, Contact data
Background
Reporting in research studies is often suboptimal [1–3]. Synthesizing the outcomes of poorly reported studies can pose a serious threat to the validity of systematic reviews and healthcare guidelines [4–7]. The Cochrane Collaboration therefore recommends reviewers to contact authors of eligible studies included in the review and to request additional information on poorly reported items in these studies [8, 9]. In new Cochrane intervention reviews, we will quantify issues such as (1) how reviewers report on procedures and outcomes of contacting authors to obtain additional information, (2) how contacted authors reply, and (3) the consequences of the additional obtained data for the risk of bias and Grading of Recommendations, Assessment, Development and Evaluation (GRADE) scores [10] and for the primary and secondary outcomes of the review.
Poor reporting does not permit an adequate assessment of how a study was conducted and whether systematic error could have influenced outcomes. Various research papers have assessed these issues and have confirmed that reporting of trial findings is often incomplete and biased [1–3]. Adverse events are also often underreported and can lead to misconceptions on the safety of interventions [11–13]. Addressing incomplete reporting of primary studies is particularly important for systematic reviews, because reporting issues can influence the conclusions of such studies [4–7] and could raise serious doubts whether the outcomes of a systematic review can be trusted. This is a problem for a variety of stakeholders including patients, clinicians, researchers, health care policymakers, insurance companies, pertinent healthcare industries, research sponsors, or others.
To systematically tackle inadequate reporting, the Enhancing the Quality and Transparency Of health Research (EQUATOR) Network seeks to improve reporting by promoting the wider use of key reporting standards for a variety of study designs [14]. Although these guidelines are endorsed by many journals, improvements of reporting in research articles are still small or insufficient [15, 16]. Contacting the original authors of a research study by systematic reviewers to obtain additional research data is another strategy for dealing with incomplete reporting [8, 9, 17]. Such methods have provided important research data that have permitted systematic reviewers to (1) assess the eligibility of studies [18, 19], (2) assess conflicts of interest [20], (3) obtain research protocols [21], (4) assess the study design [19], (5) obtain information on groups of participants [22], (6) obtain outcomes [19], (7) calculate missing statistics [18, 23, 24], (8) assess sample sizes [25], (9) assess information on blinding of participants and personnel [26], (10) fine-tune risk of bias assessments [23, 24], and (11) assess the risk of selective outcome reporting [21]. Contacting of study authors is defined as “Highly desirable” according to the Methodological standards for the conduct of new Cochrane Intervention Reviews (MECIR) [9]. To scrutinize this issue, we will ask a series of questions on contacting of authors in these “MECIR guided” systematic reviews such as (1) how the reviewers report on contacting of authors of eligible studies, (2) the prevalence of reviews that contacted studies, (3) the validity of author contact data, (4) how contacted authors reply, (5) how reviewers report on the obtained additional data, and (6) what the consequences of these additional data are for the risk of bias scores of the eligible primary studies and for the GRADE [10] scores and the primary and secondary outcomes of the review. We will also quantify the prevalence of “Unclear” risk of bias scores in the eligible studies in these new Cochrane intervention reviews, because this statistic could be an indicator of the need to contact authors of primary research studies to obtain additional information.
To assess what has been published on these issues, we conducted scoping searches in the following electronic databases: Pubmed (MEDLINE), EMBASE (Ovid), and Cochrane Central Register of Controlled Trials (CENTRAL) [27]. The “Related Articles” feature in PubMed was also used to find additional studies. These procedures identified various publications on contacting of authors, but little has been published on our research questions [17, 28–34]. For example, (1) Two papers [17, 28] discussed the importance of contacting authors, (2) Schroll et al. [29], Selph et al. [30], and Van Driel et al. [31] asked different research questions on contacting authors, (3) Wolfe et al. [32] did not ask most of our questions, focused exclusively on missing drug trial data, and used different methodology, (4) Young et al. [33] assessed the effects of different methods to obtain additional data, but did not ask our research questions (3), and (5) Mullan et al. [34] showed that procedures and outcomes of contacting authors in top medical journals and Cochrane reviews were suboptimal but this study included only a small number of Cochrane reviews (n = 54), assessed data of more than 10 years ago, and did not cover most of our research questions. We also conducted a cross-sectional and a survey pilot study in 2 months of Cochrane intervention reviews (July and August 2015), which (1) confirmed the importance of addressing our questions, (2) showed the need to use both a cross-sectional study and a survey to adequately address these questions, and (3) identified at least one “Unclear” risk of bias domain in most of the eligible studies of these reviews. The published evidence on the importance of obtaining additional research information from original investigators, the knowledge gaps in the literature on our research questions, and the outcomes of our pilot tests all demonstrated the need for undertaking this study.
Objectives
Cross-sectional study
The main objectives for our cross-sectional study are as follows:
To calculate the prevalence of reviews that reported that they contacted studies to obtain additional information
To calculate the prevalence of reviews that reported the number of all contacted studies
To calculate the prevalence of reviews that reported that all studies with at least one “Unclear” (as a result of missing or insufficient information) risk of bias score were contacted
To calculate the prevalence of reviews that reported the number of all replying studies
To calculate the prevalence of replying studies among the contacted studies
To calculate the prevalence of reviews that reported what information data was (were) obtained from each of the replying studies
To calculate the prevalence of reviews that reported the consequences of each of the obtained information data that was (were) obtained from each of the replying studies
To calculate the prevalence of eligible primary studies with at least one domain scored as “Unclear” risk of bias
Survey
The main objectives for our survey are as follows:
To calculate the prevalence of reviews that contacted studies to obtain additional information
To calculate the prevalence of reviews in which all contacted studies had valid contact data
To quantify the consequences of contacting studies on risk of bias scores, GRADE scores [10], and on the primary and secondary outcomes of the review
Cross-sectional study versus the survey
The following objective was defined:
To quantify the prevalence of reviews that reported in both the cross-sectional study and in the survey that studies were contacted to obtain additional information
Methods
We adopted the Preferred Reporting Items for Systematic review and Meta-analysis Protocols (PRISMA-P) 2015 statement and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement as the frameworks for reporting this protocol, because these guidelines cover most of our planned methods [14, 35–37]. Our procedures for developing this protocol were conducted by three operators (RMR, LL, and NDG). Differences between this protocol and the final manuscript will be presented in the completed research study together with the rationale for these changes.
Figure 1 summarizes the flow of the methods in this research study. We first pilot tested the following items on 2 months (July and August 2015) of eligible new Cochrane intervention reviews: (1) our research questions and eligibility criteria for the cross-sectional study, (2) the study selection and data extraction procedures for the cross-sectional study, (3) the survey questions and the pertinent contacting procedures (steps 1–3). These items were subsequently fine-tuned (step 4). Our pilot tests are summarized in Additional file 1. For step 5, we will select eligible systematic reviews in the Cochrane Database. The same selected eligible reviews will be used for both the cross-sectional study and the survey. To avoid the risk of foreknowledge of outcomes, we will complete all data extraction for the cross-sectional study (step 6) prior to conducting the survey (step 7). To avoid bias during data analyses (step 8), we will use Excel files that conceal all reference to the origin of the data, i.e., files without the titles of the reviews and the names and emails of the authors. In the following sections, we will first present our methods for the cross-sectional study and then our methods for the survey.
Fig. 1.
Flow diagram of the research methods
Methods for the cross-sectional study
Eligibility criteria
Type of studies:
Only Cochrane systematic reviews will be included. These reviews will be selected, because they are considered the reference standard for conducting and reporting such research studies [38].
We want to include a full year of Cochrane systematic reviews. Only Cochrane reviews published in 2016 are eligible, because this is the most current complete year.
In 2011, Cochrane working groups started to develop the “Methodological Expectations of Cochrane Intervention Reviews (MECIR)” standards to maintain the high quality of these reviews [9]. The MECIR standards have seen various updates, and until October 2016, they were specifically designed for new Cochrane intervention reviews and were not applicable for “updates” of Cochrane reviews [9, 39-41]. In this study, we will only include new Cochrane intervention reviews published in 2016 and will exclude updates, because this change of the MECIR standard was relatively recent and including updates of reviews published prior to October 2016 could introduce additional heterogeneity and bias.
Manuscripts labeled in the Cochrane library as “Protocol,” “Withdrawn Protocols,” “Methodology,” “Diagnostic,” “Overview,” “Prognosis,” “Qualitative,” “New search,” “Update,” and “Withdrawn” will be excluded. Earlier published reviews that were later split into two or more new reviews will also be excluded.
Reviews that included one or more primary studies will be eligible. Empty reviews [42], i.e., reviews that did not identify eligible studies for the research question will be excluded, because our study will focus on contacting authors of eligible studies and not on contacting authors of potentially eligible studies (see interventions).
Eligibility criteria will not be applied for specific research designs, participants, interventions, comparators, outcomes, endpoints, or settings in the primary studies of the eligible systematic reviews.
Since we will also quantify “Unclear” risk of bias scores in the eligible primary studies, we will exclude reviews that do not present the pertinent risk of bias tables with these scores.
Interventions:
Eligible interventions refer to contacting by reviewers of the authors of eligible primary studies included in the review to obtain additional research data on these studies (e.g., information on unreported or missing data, individual patient data, clarification of research methods, and bias issues).
Non-eligible interventions are (1) contacting by reviewers of authors of primary research studies to verify their eligibility for the review, (2) contacting by reviewers of researchers or research sponsors to get information on additional unknown completed or ongoing studies, and (3) contacting by reviewers of research sponsors to obtain additional research data of the eligible primary studies. We will exclude these interventions, because it will make our cross-sectional study too broad and too difficult to handle
We expect that most of the contacting procedures are done through emails, but any method for contacting authors such as sending letters, telephoning, faxing, or visiting investigators of primary research studies will also be eligible.
Outcomes:
Our statistics to address our primary objectives are based on data on contacting and replying of original investigators of eligible studies. Confusion on calculating these statistics could arise when more than one author was contacted or more than once. For the interventions in this study, we will therefore use the terms “contacting studies” and “replying studies”. We will consider each “contacted study” or each “replying study” as a single entity, even when several of its authors were contacted or replied once or several times.
Information sources and search strategy
The Cochrane Database of Systematic Reviews (CDSR) will be hand-searched for eligible systematic reviews published in this database for the year 2016 [43].
Study records
Data management
To apply our study selection procedures consistently and to reduce inter-examiner disagreements, we consulted the guidelines of the Cochrane Handbook for Systematic Reviews of Interventions and the PRISMA-P 2015 statement [8, 35, 36].
Selection of studies
Selection of studies is based on the following:
Prior to the formal study selection, we conducted pilot tests to fine-tune these procedures and to calibrate reviewers (Fig. 1)(Additional file 1).
Two reviewers (RMR and LL) will conduct the formal searches for eligible reviews published in 2016 in The Cochrane Database of Systematic Reviews (CDSR) [43]. We will search under the heading “Reviews” and will only consider intervention reviews.
We will also check the “History” section and the section “References to other published versions of this review” of each potentially eligible intervention reviews to exclude potentially “mislabeled” reviews.
Disagreements on eligibility of studies between authors will be resolved through discussions. A methodologist (NDG) will be consulted to resolve persisting disagreements.
All study selection steps will be presented in a PRISMA flow diagram [44]. References of all included studies will be listed in a table. References to excluded studies will be listed separately in a table with the rationale for their exclusion.
Data extraction and management
We will download all eligible reviews as PDFs and will divide them according to the month of publication. Our data extraction and management procedures consist of various steps which are explained under here.
Defining “contacting studies”:
“Contacting studies” refers to studies that are contacted to obtain additional information on an eligible article (e.g., unreported or missing data, individual patient data, clarification of research methods, and risk of bias issues).
The authors that are contacted in the eligible studies of the reviews can be any author of these studies.
Defining search terms:
To identify pertinent subject headings and keywords, we applied the following protocol: (1) All new intervention reviews published in the July 2015 Cochrane Library that explicitly described the methods and outcomes of contacting authors were selected by two review authors (RMR and LL). These reviews were then hand-searched to detect and fine-tune pertinent subject headings associated with procedures for contacting studies; (2) synonyms, acronyms, related terms, abbreviations, and variant spellings of these search terms were also identified. Thesauri of databases were consulted for this purpose.
These searches identified a variety of search terms and their respective grammatical derivatives which are presented in Additional file 2.
Developing data extraction strategies:
We adopted the Adobe Acrobat protocol for searching and marking multiple words in a PDF [45].
We used this protocol to assist with our hand searching of the pertinent data items in the eligible systematic reviews. We conducted three different tests to validate these data extraction strategies (Additional file 1).
We will merge all PDFs of the eligible reviews into 12 single binder PDFs. Each binder PDF will contain all eligible reviews for a specific month of the year 2016 [46]. This method was chosen, because it will reduce the risk of introducing error, because multiple search terms do not have to be re-typed in the Adobe Acrobat search box for each eligible study. Merging the PDFs of the entire year of 2016 into one single PDF was avoided, because such a document will become difficult to handle.
Developing data collection forms and defining data items:
The new Cochrane intervention reviews published in July and August 2015 were used to (1) define a broad-spectrum of data items, (2) develop and fine-tune data collection forms, and (3) pilot-test inter-operator differences in the data extraction procedures (Additional file 1).
The pilot tested data collection forms are presented in Additional file 3 and will be transferred to an Excel form for the actual data extraction.
The criteria for completing these forms were defined in detail in order to reduce the need of revisiting selected papers and to improve reproducibility and inter-operator accuracy [36, 47].
Our questions on contacting of authors for the cross-sectional study are presented in Table 1, and Fig. 2 presents the flow diagram of these questions.
Table 1.
Questions on contacting of authors for the cross-sectional study
| Section | Questions and answers and criteria for addressing questions |
|---|---|
| Reporting on contacting of studies | Question 1. Did the reviewers report that they contacted studies? |
| Answer 1. Yes/no | |
| Criteria for addressing question 1: | |
| Yes: When the reviewers report that they contacted one or more studies to obtain additional information. “Yes” is also scored when reviewers reported that they wanted to contact one or more of these studies, but contact information could not be obtained. | |
| No: When the reviewers did not report whether studies were contacted to obtain additional information. | |
| No: When reviewers report that they did not contact studies to obtain additional information. | |
| “No” is also scored when reviewers describe that they planned to contact studies in the methods section, but do not further report on contacting studies. | |
| “No” is also scored when in the section “Contribution of Authors” specific reviewers are linked to contacting authors without further specification, but in the review itself there is no reported proof that studies were actually contacted. The rationale for this “No” score is that such linking could indeed refer to contacting of studies to obtain additional information (e.g., missing data and risk of bias), but could also refer to contacting of authors to assess the eligibility of studies or contacting of authors to identify additional or ongoing studies. | |
| Reporting on contacting of studies with “Unclear” risk of bias domainsa | Question 2. Did the reviewers report that all studies with at least one “Unclear” (as a result of missing or insufficient information) risk of bias score were contacted? |
| Answer 2. Yes/no/not applicable (NA) | |
| Criteria for addressing question 2: | |
| Yes: Studies were contacted and all studies with at least one “Unclear” (as a result of missing or insufficient information) risk of bias score were contacted. We will still score “Yes” when reviewers reported that they wanted to contact all studies with “Unclear” (as a result of missing or insufficient information) risk of bias scores, but contact information could not be obtained for one or more of these studies. | |
| No: Studies were contacted, but not all studies with at least one “Unclear” (as a result of missing or insufficient information) risk of bias score were contacted. | |
| NA: Studies were contacted, but “Unclear” (as a result of missing or insufficient information) risk of bias scores were not assigned to any of the domains of the included studies. | |
| Reporting all contacted studies | Question 3. Could the number of all contacted studies be identified in the review? |
| Answer 3. Yes/no | |
| Criteria for addressing question 3: | |
| Yes: When the number of all contacted studies could be identified in the review. We still scored “Yes” when reviewers reported that they wanted to contact one or more of these studies, but contact information could not be obtained. | |
| No: When the number of all contacted studies could not be identified in the review. | |
| No: When one or more studies have been contacted, but it was impossible to identify the exact total number of studies that were contacted. | |
| Number of contacted studies | Question 4. What is the number of contacted studies in the review? |
| Answer 4. Present the number of contacted studies in the review. | |
| Criteria for addressing question 4: | |
| Only the actual number of studies that was contacted will be scored. Studies that were not contacted because contact information was not available will not be included in this number. | |
| Reporting all replying studies | Question 5. Could the number of all replying studies be identified in the review? |
| Answer 5. Yes/no | |
| Criteria for addressing question 5: | |
| Yes: When the number of all replying studies could be identified in the review. | |
| No: When the number of all replying studies could not be identified (e.g., as a result of unclear reporting) in the review. | |
| No: When one or more contacted studies replied, but it was impossible to identify the exact total number of studies that replied. | |
| Number of replying studies | Question 6. What is the number of contacted studies in the review that replied? |
| Answer 6. Present the number of studies that replied. | |
| Criteria for addressing question 6: | |
| The actual number of studies that replied will be scored. | |
| Reporting on obtained additional information data | Question 7. Did the reviewers report what information data was(were) obtained from each of the replying studies? |
| Answer 7. Yes/no | |
| Criteria for addressing question 7: | |
| Yes: The reviewers reported what information data was(were) obtained from each of the replying studies. | |
| Yes: The reviewers reported that the replying studies explained that they could not provide the requested data. | |
| No: The reviewers did not report what information data was(were) obtained from each of the replying studies. | |
| No: The reviewers reported that information data was(were) obtained from the replying studies, but this was partially reported or it was unclear what these data were. | |
| Reporting on the consequences of obtained additional information data | Question 8. Were the consequences (e.g., modified statistics and risk of bias or GRADE scores) of each of the obtained information data reported? |
| Answer 8. Yes/no | |
| Criteria for addressing question 8: | |
| Yes: The review reported the consequences of each of the obtained information data. | |
| No: The review did not report the consequences of any of the obtained information data. | |
| No: The review reported the consequences of some obtained information data, but not for each of the obtained information data. |
aHiggins et al. [47] divide the definition of “Unclear” risk of bias in three subgroups: Studies are assessed as at unclear risk of bias (1) when too few details are available to make a judgement of “high” or “low” risk; (2) when the risk of bias is genuinely unknown despite sufficient information about the conduct; or (3) when an entry is not relevant to a study (for example because the study did not address any of the outcomes in the group of outcomes to which the entry applies). In this cross-sectional study, we only refer to contacting of authors for the first subgroup of the definition of “Unclear” risk of bias by Higgins et al. [47].
Fig. 2.
Flow diagram of the research questions on contacting of authors in the cross-sectional study. NA not applicable
Extracting data from reports:
Because our pilot testing showed the reliability of the Adobe Acrobat protocol for the identification of the pertinent multiple search terms, we will implement this search tool for all our data extraction procedures during our manual searches [45].
This protocol will not be applied to just the text of the eligible systematic reviews, but will cover the entire Cochrane review including all figures, tables, and references.
All research data will be extracted independently by two reviewers (RMR and LL). Disagreements on extracted data between these operators will be resolved through discussion. A third operator (NDG) will be consulted in the case of persisting disagreements.
To facilitate these discussions, both reviewers will highlight all pertinent items on contacting studies in “yellow” in the binder PDFs.
When during the data extraction procedures additional relevant data items are identified, we will include these items in the data collection forms and explain the rationale for collecting them and will report this information in the section “Differences between protocol and review”.
Methods for the survey
Our cross-sectional pilot study of 2 months of new Cochrane intervention reviews (July and August 2015) showed that various items could not be extracted from these reviews, because they were underreported. We therefore decided to include a survey to extract these underreported items. We will use the same reviews that are eligible for the cross-sectional study, i.e., new Cochrane intervention reviews published in 2016 also for our survey. For developing our survey questions, we consulted the guidelines outlined by Fowler [48] and the Checklist for Reporting Results of Internet E Surveys (CHERRIES) [49, 50]. The survey was pilot-tested on the same 2 months of new Cochrane intervention reviews as those used for the cross-sectional study.
Our survey questions on contacting authors by systematic reviewers are presented in Table 2. We will use Google Forms as the platform for our survey [51]. We created an introductory email for the survey and a landing page with an abridged protocol to provide the surveyees with additional information on our survey (Additional file 4). We consulted our published protocols for contacting authors of primary research studies to fine-tune our contacting procedures [52–55].
Table 2.
Questions on contacting of authors for the survey
| Section | Questions and answers and criteria for addressing questions |
|---|---|
| Title of the systematic review | Question 1. Could you please insert the title of your review? |
| (Please copy and paste your review title from our email) | |
| Answer 1. Insert title: …………………………………………………………………………………….. | |
| Contacting of the eligible included studies for additional information | Question 2. Did you contact any of the study authors of the eligible studies in your review to obtain additional information (e.g., issues on risk of bias and unclear or missing data) on these included studies? |
| Please assess carefully the possible answers to this question: | |
| Yes: If you contacted (through mail, telephone etc.) one or more authors of the eligible studies in your review to obtain additional information on these included studies (e.g., issues on risk of bias, unclear, or missing data). | |
| No: When authors were not contacted or when authors were contacted for other reasons e.g., to verify potential eligibility of studies or to obtain information on unknown completed or ongoing studies. | |
| Answer 2. Yes or no | |
| Valid contact data | Question 3. Had all contacted studies valid contact data? |
| Please assess carefully the possible answers to this question: | |
| Valid contact data refers to having one or more of the following functioning contact data, i.e., email addresses, telephone numbers, fax numbers, and postal addresses. | |
| Answer 3. Yes or no | |
| Effect on risk of bias scores | Question 4. Were one or more risk of bias scores in your review modified as a result of the information obtained from contacted studies? |
| Please assess carefully the possible answers to this question: | |
| Yes: One or more risk of bias scores in your review were modified as a result of the information obtained from the contacted studies. | |
| No: None of the risk of bias scores in your review were modified as a result of the information obtained from the contacted studies. | |
| Not applicable: Risk of bias was not assessed in your review or this question could not be addressed for other reasons. | |
| Answer 4. Yes, no, not applicable | |
| Effect on the GRADE score | Question 5. Were GRADE scores for one or more outcomes modified as a result of the information obtained from the contacted studies? |
| Please assess carefully the possible answers to this question: | |
| Yes: GRADE scores for one or more outcomes were modified as a result of the information obtained from the contacted studies. | |
| No: GRADE scores for one or more outcomes were not modified as a result of the information obtained from the contacted studies. | |
| Not applicable: GRADE scores were not assessed in your review or this question could not be addressed for other reasons. | |
| Answer 5. Yes, no, not applicable | |
| Effect on the summary primary or secondary outcomes | Question 6. Were any of the summary primary or secondary outcomes of the review modified as a result of the information obtained from the contacted studies? |
| Please assess carefully the possible answers to this question: | |
| Yes: One or more summary primary or secondary outcomes of the review were modified as a result of the information obtained from the contacted studies. | |
| No: None of the summary primary or secondary outcomes of the review were modified as a result of the information obtained from the contacted studies. | |
| Answer 6. Yes or no | |
| Effect on the summary effect size of the primary outcome | Question 7. How was the summary effect size of the primary outcome modified as a result of the information obtained from the contacted studies? |
| Please assess carefully the possible answers to this question: | |
| Increased: The summary effect size of the primary outcome increased as a result of the information obtained from the contacted studies. | |
| Decreased: The summary effect size of the primary outcome decreased as a result of the information obtained from the contacted studies. | |
| Unchanged: The summary effect size of the primary outcome did not change as a result of the information obtained from the contacted studies. | |
| Not applicable: The summary effect size of the primary outcome was not measured in your review or this question could not be addressed for other reasons. | |
| Answer 7. Increased, decreased, unchanged, not applicable | |
| Optional question | Question 8. Could you please describe some issues that you want to share with us regarding contacting of study authors of the eligible studies included in your review? |
| Answer 8. I have the following issues to share: | |
| I have no further issues to share. |
Our procedures for conducting our survey are presented as a step-by-step process:
Step 1. The titles, names of the authors, and the contact addresses of the eligible reviews will be extracted and copied and pasted into an Excel data sheet.
Step 2. Our survey will be sent with an introductory email to the contact address of each eligible review. We will only send the survey to the contact address of the eligible reviews and not to any of the co-authors. When more than one email address was provided for the same contact author, we will send the survey to both addresses. In the initial email, authors will answer our survey through a link presented in this email. In all reminder emails, we give authors the option to either answer the survey through a link or simply by completing the survey presented at the end of the introductory email.
Step 3. In the first week of our survey, we will send our initial email and our first reminder email (in the case that no response will be received to our initial email). We will send this first reminder email after a minimum of three business days [30].
Step 4. When still no response will be received, we will continue to send reminder emails during the following 2 weeks (two reminders per week). In each reminder email, we will indicate the current response rate to our survey. After a maximum of five reminder emails, we will stop the survey. In our fifth reminder email, we will indicate that it is the last email for our survey.
When automated responses like “out of office,” “vacation leave,” maternity leave, and study leave will be received, we will apply our protocol at the author’s return.
Primary outcome and power calculation
Our primary outcome for this research study will be the prevalence of reviews that reported that eligible primary studies were contacted to obtain additional information. For this statistic, the following calculation is made: the number of reviews that reported that they contacted studies to obtain additional information/the total number of eligible reviews. To calculate the required sample size, we conducted a power calculation using software developed by EpiTools epidemiological calculators [56]. We first conducted a pilot test on all eligible Cochrane systematic reviews published in July and August 2015 to identify the estimated prevalence for the primary outcome. This pilot study showed that 40 of 52 (76.92%) eligible reviews were contacted to obtain additional information. EpiTools calculated a sample size of 273 for the estimated prevalence of 0.7692 with a 0.95 confidence level (desired precision of estimate 0.05) [56].
Since 2 months (July and August 2015) of new Cochrane intervention reviews provided 52 eligible reviews, this would account for 26 reviews per months. We decided to include a complete year (2016) of new Cochrane intervention reviews, i.e., 312 reviews (12 × 26). When 12 months of new Cochrane intervention reviews will not provide the required sample size of 273 reviews, we will include subsequent months of eligible reviews, i.e., from January 2017 onwards until the desired sample size will be reached.
Outcome measures and statistical analysis
The planned outcomes for the cross-sectional study and the survey will be presented in summary of findings tables (Additional file 5). We will also compare outcomes measured in the cross-sectional study with those in the survey (Additional file 5). Outcomes that will be introduced or eliminated post hoc will be reported in the completed study together with the rationale for including or excluding them. Stata software (Stata Corporation, College Station, Texas, USA) version 15 will be used for the statistical analysis [57]. Prevalence data will be reported with their 0.95 confidence levels.
Differences between the protocol and the completed study
Differences between the protocol and the completed study will be presented, and the rationale for these modifications will also be reported. The consequences of these modifications on the magnitude, direction, and the validity of outcomes will also be given [58].
Discussion
Systematic reviewers address focused questions by appraising and synthesizing all relevant research evidence. The quality of these reviews strongly depends on the quality of the reporting of the eligible primary studies. Contacting authors of these studies to obtain additional information could be an important research procedure for dealing with suboptimal reporting. This study will explore issues such as (1) how reviewers report on contacting of authors of eligible studies in the review and the outcomes of these procedures, (2) the validity of contact data in contacted studies, (3) how contacted investigators reply, (4) how reviewers report on the obtained additional data, (5) what the consequences of these contacting procedures are for outcomes and quality scores in the review, (6) the differences between outcomes reported in the review with those recorded in the survey, and (7) the prevalence of primary studies with at least one domain scored as “Unclear” risk of bias.
Positive findings are important, because little has been published on these research questions and such outcomes could address some knowledge gaps and could quantify the importance of contacting of authors. However, negative outcomes are expected, because our pilots test showed (1) that procedures of contacting authors and the consequences are underreported in systematic reviews of interventions and (2) that contacted authors of primary studies reply poorly to reviewers. These negative findings could show the need for developing guidelines for reporting and conducting of contacting procedures of primary research studies and their outcomes in systematic reviews. These findings could direct research on this topic and could also show the need to keep contact data of investigators up to date in a central register. Better methods of conducting and reporting of communication between systematic reviewers and authors of primary research studies could lead to a better understanding of what was done in the eligible studies and could ultimate improve the quality and the trustworthiness of the review and its reproducibility. Patients, clinicians, researchers, guideline developers, research sponsors, and the general public will all be beneficiaries.
Additional files
Pilot tests. (DOCX 42 kb)
Search terms and their derivatives. (DOCX 37 kb)
Data collection forms for the cross-sectional study. (DOCX 43 kb)
Introductory initial email and abridged protocol for the survey. (DOCX 46 kb)
Summary of findings tables. (DOCX 42 kb)
Acknowledgements
The authors wish to extend special thanks to Rossella Bassi for her assistance with the administration of the pilot studies for both the cross-sectional study and the survey.
Funding
All expenses for preparing this protocol were paid evenly by each author.
Availability of data and materials
Not applicable
Abbreviations
- EQUATOR Network
Enhancing the Quality and Transparency Of health Research Network
- MECIR
Methodological Expectations of Cochrane Intervention Reviews
Authors’ contributions
RMR and NDG conceived and designed the study protocol for both the cross-sectional study and the survey. RMR is the guarantor. RMR and LL conducted the pilot testing of the study selection procedures and data extraction forms of the cross-sectional study. RMR, NDG, and LL fine-tuned the research protocol after the pilot testing of both the cross-sectional study and the survey. All authors read and approved the final protocol.
Ethics approval and consent to participate
Not applicable
Consent for publication
Not applicable
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Footnotes
Electronic supplementary material
The online version of this article (10.1186/s13643-017-0643-z) contains supplementary material, which is available to authorized users.
Contributor Information
Reint Meursinge Reynders, Phone: 39-02-48008585, Email: reyndersmail@gmail.com.
Luisa Ladu, Email: luisaladu@gmail.com.
Nicola Di Girolamo, Email: nicoladiggi@gmail.com.
References
- 1.Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457–2465. doi: 10.1001/jama.291.20.2457. [DOI] [PubMed] [Google Scholar]
- 2.Dwan K, Gamble C, Williamson PR, Kirkham JJ; Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLoS One. 2013;8(7):e66844. [DOI] [PMC free article] [PubMed]
- 3.Dwan K, Altman DG, Clarke M, Gamble C, Higgins JP, Sterne JA, Williamson PR, Kirkham JJ. Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLoS Med. 2014;11(6):e1001666. doi: 10.1371/journal.pmed.1001666. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Bleakley CM, McDonough SM, MacAuley DC. Some conservative strategies are effective when added to controlled mobilization with external support after acute ankle sprain: a systematic review. Aust J Physiother. 2008;54(1):7–20. doi: 10.1016/S0004-9514(08)70061-8. [DOI] [PubMed] [Google Scholar]
- 5.Gordon M, Findley R. Educational interventions to improve handover in health care: a systematic review. Med Educ. 2011;45(11):1081–1089. doi: 10.1111/j.1365-2923.2011.04049.x. [DOI] [PubMed] [Google Scholar]
- 6.Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365. doi: 10.1136/bmj.c365. [DOI] [PubMed] [Google Scholar]
- 7.Meuffels DE, Reijman M, Scholten RJ, Verhaar JA. Computer assisted surgery for knee ligament reconstruction. Cochrane Database Syst Rev. 2011;6:CD007601. doi: 10.1002/14651858.CD007601.pub2. [DOI] [PubMed] [Google Scholar]
- 8.Higgins JPT, Deeks JJ. Chapter 7: selecting studies and collecting data. In: JPT H, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated march 2011): The Cochrane Collaboration; 2011. [online] Available from: http://handbook-5-1.cochrane.org/. (accessed July 28th 2017).
- 9.Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological Expectations of Cochrane Intervention Reviews (MECIR). Standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates. Cochrane: London, 2016. [online] Available from: http://methods.cochrane.org/sites/default/files/public/uploads/mecir_printed_booklet_final.pdf (accessed July 28th 2017).
- 10.Schünemann H, Brozek J, Gyatt G, Oxman A, editors. GRADE handbook. Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach. [updated October 2013]. [online] Available from: http://gdt.guidelinedevelopment.org/app/handbook/handbook.html (accessed July 28th 2017).
- 11.Ioannidis JP. Adverse events in randomized trials: neglected, restricted, distorted, and silenced. Arch Intern Med. 2009;169:1737–1739. doi: 10.1001/archinternmed.2009.313. [DOI] [PubMed] [Google Scholar]
- 12.Sigh S, Loke YK. Drug safety assessment in clinical trials: methodological challenges and opportunities. Trials. 2012;13:138. doi: 10.1186/1745-6215-13-138. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Zorzela L, Golder S, Liu Y, Pilkington K, Hartling L, Joffe A, Loke Y, Vohra S. Quality of reporting in systematic reviews of adverse events: systematic review. BMJ. 2014;348:f7668. doi: 10.1136/bmj.f7668. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Equator (Enhancing the Quality and Transparency Of health Research) Network, 2017. [online] Available from: http://www.equator-network.org (accessed July 28th 2017).
- 15.Turner L, Shamseer L, Altman DG, Schulz KF, Moher D. Does use of the CONSORT statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev. 2012;1:60. doi: 10.1186/2046-4053-1-60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, Dias S, Schulz KF, Plint AC, Moher D. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev. 2012;11:MR000030. doi: 10.1002/14651858.MR000030.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Berstock J, Beswick A. Importance of contacting authors for data on adverse events when compiling systematic reviews. BMJ. 2014;348:g1394. doi: 10.1136/bmj.g1394. [DOI] [PubMed] [Google Scholar]
- 18.Blackwood B, Burns KE, Cardwell CR, O’Halloran P. Protocolized versus non-protocolized weaning for reducing the duration of mechanical ventilation in critically ill adult patients. Cochrane Database Syst Rev. 2014;11:CD006904. doi: 10.1002/14651858.CD006904.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Van Dessel N, den Boeft M, van der Wouden JC, Kleinstäuber M, Leone SS, Terluin B, Numans ME, van der Horst HE, van Marwijk H. Non-pharmacological interventions for somatoform disorders and medically unexplained physical symptoms (MUPS) in adults. Cochrane Database Syst Rev. 2014;11:CD011142. doi: 10.1002/14651858.CD011142.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Anderson DE, Kew KM, Boyter AC. Long-acting muscarinic antagonists (LAMA) added to inhaled corticosteroids (ICS) versus the same dose of ICS alone for adults with asthma. Cochrane Database Syst Rev. 2015;8:CD011397. doi: 10.1002/14651858.CD011397.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Bromley R, Weston J, Adab N, Greenhalgh J, Sanniti A, McKay AJ, Tudur Smith C, Marson AG. Treatment for epilepsy in pregnancy: neurodevelopmental outcomes in the child. Cochrane Database Syst Rev. 2014;10:CD010236. doi: 10.1002/14651858.CD010236.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Aabenhus R, Jensen JU, Jørgensen KJ, Hróbjartsson A, Bjerrum L. Biomarkers as point-of-care tests to guide prescription of antibiotics in patients with acute respiratory infections in primary care. Cochrane Database Syst Rev. 2014;11:CD010130. doi: 10.1002/14651858.CD010130.pub2. [DOI] [PubMed] [Google Scholar]
- 23.Meursinge Reynders R, Ladu L, Ronchi L, Di Girolamo N, de Lange J, Roberts N, Plüddemann A. Insertion torque recordings for the diagnosis of contact between orthodontic mini-implants and dental roots: a systematic review. Syst Rev. 2016;5(1):50. doi: 10.1186/s13643-016-0227-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Meursinge Reynders R, Ronchi L, Ladu L, Di Girolamo N, De Lange J, Roberts N, Mickan S. Barriers and facilitators to the implementation of orthodontic mini-implants in clinical practice: a systematic review. Syst Rev. 2016;5(1):163. doi: 10.1186/s13643-016-0336-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Hirst C, Owusu-Ofori S. Prophylactic antibiotics for preventing pneumococcal infection in children with sickle cell disease. Cochrane Database Syst Rev. 2014;11:CD003427. doi: 10.1002/14651858.CD003427.pub3. [DOI] [PubMed] [Google Scholar]
- 26.Hoving JL, Lacaille D, Urquhart DM, Hannu TJ, Sluiter JK, Frings-Dresen MH. Non-pharmacological interventions for preventing job loss in workers with inflammatory arthritis. Cochrane Database Syst Rev. 2014;11:CD010208. doi: 10.1002/14651858.CD010208.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: JPT H, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated march 2011): The Cochrane Collaboration; 2011. [online] Available from: http://handbook-5-1.cochrane.org/. (accessed July 28th 2017).
- 28.Atherton H, Sawmynaden P, Car J. The value of author contact in determining risk of bias in included studies in a systematic review. Cochrane Colloquium Abstracts. [online] Available from: http://abstracts.cochrane.org/2013-qu%C3%A9bec-city/value-author-contact-determining-risk-bias-included-studies-systematic-review (accessed July 28th 2017).
- 29.Schroll JB, Bero L, Gøtzsche PC. Searching for unpublished data for Cochrane reviews: cross sectional study. BMJ. 2013;346:f2231. doi: 10.1136/bmj.f2231. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Selph SS, Ginsburg AD, Chou R. Impact of contacting study authors to obtain additional data for systematic reviews: diagnostic accuracy studies for hepatic fibrosis. Syst Rev. 2014;3:107. doi: 10.1186/2046-4053-3-107. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. J Clin Epidemiol. 2009;62(8):838–844.e3. doi: 10.1016/j.jclinepi.2008.09.010. [DOI] [PubMed] [Google Scholar]
- 32.Wolfe N, Gøtzsche PC, Bero L. Strategies for obtaining unpublished drug trial data: a qualitative interview study. Syst Rev. 2013;2:31. doi: 10.1186/2046-4053-2-31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Young T, Hopewell S. Methods for obtaining unpublished data. Cochrane Database Syst Rev. 2011;11:MR000027. doi: 10.1002/14651858.MR000027.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Mullan RJ, Flynn DN, Carlberg B, Tleyjeh IM, Kamath CC, LaBella ML, Erwin PJ, Guyatt GH, Montori VM. Systematic reviewers commonly contact study authors but do so with limited rigor. J Clin Epidemiol. 2009;62(2):138–142. doi: 10.1016/j.jclinepi.2008.08.002. [DOI] [PubMed] [Google Scholar]
- 35.Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1. doi: 10.1186/2046-4053-4-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Shamseer L, Moher D, Clarke M, Ghersi D, Deceased LA, Petticrew M, Shekelle P, Stewart LA, the PRISMA-P Group Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349:g7647. doi: 10.1136/bmj.g7647. [DOI] [PubMed] [Google Scholar]
- 37.von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, STROBE Initiative The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;147(8):573–577. doi: 10.7326/0003-4819-147-8-200710160-00010. [DOI] [PubMed] [Google Scholar]
- 38.Smith R. The Cochrane collaboration at 20. BMJ. 2013;347:f7383. doi: 10.1136/bmj.f7383. [DOI] [PubMed] [Google Scholar]
- 39.Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological Expectations of Cochrane Intervention Reviews (MECIR). Standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates. What’s new in MECIR 2016. London: Cochrane; 2016. [Google Scholar]
- 40.Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. MECIR Manual. Version 1.04. Last update October 2017. [online] Available from: http://community.cochrane.org/mecir-manual. Accessed 30 Nov 2017.
- 41.Cochrane Methods. Update July 2017. [online] Available from: http://methods.cochrane.org/mecir. Accessed 30 Nov 2017.
- 42.Yaffe J, Montgomery P, Hopewell S, Shepard LD. Empty reviews: a description and consideration of Cochrane systematic reviews with no included studies. PLoS One. 2012;7(5):e36626. doi: 10.1371/journal.pone.0036626. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Cochrane Database of Systematic Reviews (CDSR). [online] Available from: http://www.cochranelibrary.com/cochrane-database-of-systematic-reviews/ (accessed July 28th 2017).
- 44.Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6:e1000097. [PMC free article] [PubMed]
- 45.Acrobat for legal professionals. Searching and marking multiple words. [online] Available from: http://blogs.adobe.com/acrolaw/2010/04/searching-and-marking-multiple-words-in-a-pdf/ (accessed July 28th 2017).
- 46.Merge PDFs, combine files into one PDF. [online] Available from: http://www.wikihow.com/Merge-PDF-Files (accessed July 28th 2017).
- 47.Higgins JPT, Altman DG, Sterne JAC. Chapter 8: assessing risk of bias in included studies. In: JPT H, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011): The Cochrane Collaboration; 2011. [online] Available from: www.cochrane-handbook.org. (accessed October 13th 2017).
- 48.Fowler FJ., Jr . Chapter 6: designing questions to be good measures. In: Fowler FJ Jr, editor. Survey research methods (5th edition) Los Angeles (USA): SAGE publications; 2014. [Google Scholar]
- 49.Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) J Med Internet Res. 2004;6(3):e34. doi: 10.2196/jmir.6.3.e34. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Eysenbach G. Correction: improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) J Med Internet Res. 2012;14(1):e8. doi: 10.2196/jmir.2042. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Google Forms. [online] Available from: https://www.google.com/forms/about/ (accessed July 28th 2017).
- 52.Di Girolamo N, Giuffrida M, Winter A, Meursinge RR. In veterinary trials reporting and communication regarding randomization procedures is suboptimal. Vet Rec. 2017; 10.1136/vr.104035. [DOI] [PubMed]
- 53.Meursinge Reynders R, Ronchi L, Ladu L, Van Etten-Jamaludin F, Bipat S. Insertion torque and orthodontic mini-implants: a systematic review of the artificial bone literature. Proc Inst Mech Eng H. 2013;227(11):1181–1202. doi: 10.1177/0954411913495986. [DOI] [PubMed] [Google Scholar]
- 54.Meursinge Reynders R, Ladu L, Ronchi L, Di Girolamo N, De Lange J, Roberts N, Plüddemann A. Insertion torque recordings for the diagnosis of contact between orthodontic mini-implants and dental roots: protocol for a systematic review. Syst Rev. 2015;4(1):39. doi: 10.1186/s13643-015-0014-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Meursinge Reynders R, Ronchi L, Ladu L, Di Girolamo N, De Lange J, Roberts N, Mickan S. Barriers and facilitators to the implementation of orthodontic mini-implants in clinical practice: a protocol for a systematic review and meta-analysis. Syst Rev. 2016;5(1):22. doi: 10.1186/s13643-016-0198-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Sergeant ESG. Epitools epidemiological calculators. Ausvet Pty Ltd. 2017. [online] Available from: http://epitools.ausvet.com.au. (accessed July 28th 2017).
- 57.StataCorp . Stata statistical software: release 15. College Station, TX: StataCorp LLC; 2017. [Google Scholar]
- 58.Higgins JPT, Deeks JJ, Altman DG (editors). Chapter 16: Special topics in statistics. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. [online] Available from: http://handbook-5-1.cochrane.org/ (accessed July 28th 2017).
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Pilot tests. (DOCX 42 kb)
Search terms and their derivatives. (DOCX 37 kb)
Data collection forms for the cross-sectional study. (DOCX 43 kb)
Introductory initial email and abridged protocol for the survey. (DOCX 46 kb)
Summary of findings tables. (DOCX 42 kb)
Data Availability Statement
Not applicable