Quality of abstracts of pilot trials in heart failure: A protocol for a systematic survey

Abstract

Introduction

Pilot trials are initial small-scale studies done to inform the design of larger trials. Their findings like other studies are usually disseminated as peer-reviewed journal articles. Abstracts are used to introduce the contents to readers, and give a general idea about the full reports and sometimes are the only source of information available to readers. Despite their importance, the contents of abstracts of trial reports are usually not informative enough and lack the essential details.

Methods and analysis

This is a protocol for a planned systematic survey with a primary aim of analyzing the reporting quality measured as the completeness of the reporting of pilot trial abstracts in heart failure. The secondary aim will be to explore factors associated with better reporting quality.

Abstracts of heart failure pilot trials in humans (journal and conference abstracts) published in the English language from 1 January 1990 to 30 November 2016 will be assessed to determine the reporting quality, based on the CONSORT 2010 statement extension to randomized pilot and feasibility trials. All non-pilot/feasibility trials and non-human pilot trials will be excluded. We will search Medline (PUBMED), Cochrane controlled trials register, Scopus and African wide information databases for pilot trials in heart failure. Title and abstracts of identified studies will be screened for inclusion and data extracted independently by two reviewers in duplicate without using the full text. Reported and unreported items on the abstracts will be presented as frequencies and percentages, a descriptive analysis will be used to interpret the reporting quality and regression analysis used for characteristics associated with greater statistical reporting at 95% confidence interval.

Review registration number

PROSPERO CRD42016049911.

1. Introduction

Abstracts are introductory summaries of full text that provide readers with a quick overview of the contents of the papers. They serve as an important aid in knowledge dissemination. Many researchers rely on abstracts as a concise source of information [1], helping them follow developments in the literature [2] and in reaching decisions on what articles to read in detail. Reasons advanced for the essential role of abstracts include the challenge of the large volume of literature, unavailability/inaccessibility of full text to some readers due to high cost [3], [4], and the fact that some articles not published in English provide abstracts in English to reach a wider audience. These important roles of abstracts, therefore, require that they contain sufficient and accurate information that will guild the readers into contents of the full text [5].

Pilot trials refer to initial small studies that researchers use in reaching the decision on commencing larger confirmatory trials [6], [7], [8]. They are comprised of a distinctive group of randomized controlled trials (RCT) often referred to as pilot and feasibility trials which do not have effectiveness or efficacy as their primary focus [9]. By the nature of their design, they are not powered for hypothesis testing, but rather should emphasize on confidence interval estimation and are usually designed to support the development of a future definitive RCT.

The consolidated standard for reporting of trials (CONSORT) statement (www.consort-statement.org) originated because of the concern raised over the years on the quality of report of RCTs [10]. It is a guideline that was designed to improve the quality of reporting, first published in 1996 revised in 2001 and later updated in 2010 [11], [12], to address this observation by providing a benchmark for complete reporting of trials. This development has been well received by many peer review journals [13] and has been shown to improve the quality of reporting of trials [14], [15], [16].

The noted improvement has led to the extension of the checklist to various other forms of trials such as non-inferiority, equivalence, and cluster or pragmatic designs. There have also been extensions for different types of interventions (non-drug treatments and herbal interventions), Patient-reported outcomes, as well as extensions for reporting harms. One major characteristic of the main CONSORT statement and all the current extensions is that they focus on trials for which the research question centers on the effectiveness or efficacy of an intervention. But as stated above pilot and feasibility trials are designed for a different purpose, serving as a precursor to definitive trials and have been at various times described as a neglected arm of medical research [17]. In order to improve reporting of pilot and feasibility trials, a group of researchers recently developed a checklist extension to serve this function [18].

In this paper, we present a protocol for a systematic survey of quality of reporting in abstracts of pilot trials in heart failure, aimed at evaluating the completeness of such reports based on the checklist extension. For the purpose of the survey, heart failure is defined as a clinical condition often referred to as congestive heart failure (CHF), which occurs when the heart is unable to pump sufficiently to maintain blood flow to meet the body's needs [19]. We will be reviewing abstracts of journal publications and conferences focusing on all types of heart failure. We chose heart failure for this survey because of its role in the global burden of cardiovascular diseases [20], [21] and the large volumes of trials in heart failure.

In the last two decades, an increasing number of clinical trials in cardiovascular disease (CVD) have been conducted mainly due to the rising prominence of CVD as one of the leading causes of morbidity and mortality [21]. A sizable number of these clinical studies were preceded by pilot trials and their results disseminated to the public through publications. However, these pilot studies were not exempt from the inconsistency in quality reporting of most of the randomized control trials (RCT). The quality of reporting across journals is variable with some journal abstracts communicating adequate information and some grossly insufficient for accurate interpretation [22]. The correct interpretation of abstracts can be enhanced if the reporting of the study design, methods, and results are complete and uniform across journals [23], [24]. Similarly, a structured and detailed reporting of RCT helps guideline developers and policy makers as they rely on RCTs [24], [25]. Incomplete information which may follow using a very small sample or not well-defined outcomes [26], [27] make it difficult to trust the findings resulting in suboptimal use of these RCTs [28]. Other factors that have also been reported to influence the quality of report of trials include multi-center studies, trials involving pharmacological studies, industrial sponsored studies and those reporting positive results for their primary outcome [29]. Hence, it is imperative for authors to report complete details of their research and for journals to ensure proper reporting is adhered to by authors.

The primary hypothesis in the planned systematic survey is that the quality of reports in abstracts of pilot trials in heart failure based on the CONSORT checklist extension for pilot and feasibility trials is poor. The exploratory hypothesis is that items in CONSORT extension for reporting of pilot trials will be seen more in publications that contain the characteristics mentioned above.

2. Methods and analysis

2.1. Primary Objective

To evaluate the reporting quality of abstracts of pilot trials in heart failure in the past 26 years (1990–2016), using CONSORT extension for reporting of abstracts of pilot trials as the reference standard.

2.2. Secondary Objectives

• 1To identify aspects of the checklist consistently reported in pilot trials
• 2To identify factors associated with proper reporting of abstracts of pilot trials

2.3. Inclusion criteria

• •Type of studies: pilot/feasibility heart failure trial with a randomized control design (parallel or cluster) done in humans,
• •Participants: reports on population with heart failure,
• •Intervention: pharmacological and non-pharmacological interventions evaluating clinical outcomes,
• •Publications from 1st January 1990 to 30 November 2016.

2.4. Exclusion criteria

• •Non-pilot trials in heart failure,
• •Animal studies.

2.5. Study design

The proposed survey will involve a systematic review of a sample of abstracts from all available publications on pilot trials in heart failure published from January 1990 to November 2016. We will systematically search Medline (PUBMED), Cochrane controlled trial register, Scopus, and African wide information electronic databases using search terms and medical subject headings (MESH) to identify heart failure pilot trials investigating pharmacological and non-pharmacological interventions (Table 1 and Appendix A). Articles will be restricted to those written in the English language.

Table 1.

Medline search strategy.

(((((Heart failure[MeSH Terms]) OR ((Heart failure OR paroxysmal dyspnea OR diastolic heart failure OR systolic heart failure OR Cardiac Failure OR Heart Decompensation OR Myocardial Failure OR Congestive Heart Failure OR Ventricular dysfunction OR cardiac insufficiency OR myocardial failure OR myocardial insufficiency)))) AND ((randomized controlled trial [pt] OR controlled clinical trial [pt] OR randomized controlled trials [mh] OR random allocation [mh] OR double-blind method [mh] OR single-blind method [mh] OR clinical trial [pt] OR clinical trials [mh] OR ("clinical trial" [tw]) OR ((singl* [tw] OR doubl* [tw] OR trebl* [tw] OR tripl* [tw]) AND (mask* [tw] OR blind* [tw])) OR (placebos [mh] OR placebo* [tw] OR random* [tw] OR research design [mh:noexp] OR (comparative study) OR (comparative studies) OR (evaluation studies) OR (evaluation study) OR follow-up studies [mh] OR prospective studies [mh] OR controlled [tw]OR controls [tw]OR control [tw] OR prospectiv* [tw] OR volunteer* [tw]) NOT (animals [mh] NOT human [mh]))))) AND (((Pilot projects[MeSH Terms]) OR Feasibility Studies[MeSH Terms]) OR ((pilot OR feasibility)))

3. Data extraction and synthesis

Selected articles based on the key search words of pilot trials in heart failure from 1990 to 2016 will be exported to Endnote X7^® reference manager software; duplicates will be identified and removed. Screening and data abstraction will be done independently and in duplicate by two reviewers (GI and MZ) using a customized data extraction form in Microsoft Excel^® format. Reviewer agreements will be measured using kappa statistics [30]. Discrepancies and differences of opinion will be resolved by consensus between reviewers or arbitration by a third reviewer. The abstracts (both journal and conference) will be evaluated using the 16 items of the CONSORT extension (Appendix B); where the item combines two or more pieces of information (e.g. eligibility and setting), each will be counted separately if it can stand alone. Where they are open to interpretation, the reviewers' interpretation of relevant information will be used. Reported items will be checked as yes and unreported items as no. Non-applicable items will be checked as NA. All “yes” responses will be assigned the number 1, while “no” and NA will be assigned 0. The overall quality of the abstracts will be calculated as a proportion of yes.

3.1. General characteristics

Information to extract from the publications will include names and addresses of authors, journal names, impact factor, Journal policy on the endorsement of CONSORT statement (by checking on the website of the journal), year of publication.

3.2. Assessment of abstract using CONSORT extension for pilot and feasibility trials checklist

Title identification as trials will be checked, and abstract format noted as structured or unstructured. Trial design (Cluster/parallel), participant's characteristics such as eligibility and setting of pilot trial conduct, type of intervention in each group, Single or multiple centers, as well as defined pre-specified objectives. Other things to be checked will include pre-specified measurements to determine the outcome, method of randomization and blinding, the number of participants screened and randomized. There will also be an evaluation of analysis done in each group including study outcome, explanation of harm reporting and general interpretation of result discussing risk and benefit. Finally, we will check the conclusions drawn from the study, implications for future definitive trial, trial registration and funding information.

4. Definition of adequate reporting

The CONSORT extension for reporting of abstracts has 16 items which are expected to be in an abstract. In this survey, a publication abstract will be judged as adequate if all the items are reported.

4.1. Sampling scheme and sample size calculation

We will consider for inclusion in the systematic survey, all sampled studies described as pilot or feasibility randomized control trials (RCT) from the general sampling population of RCTs in heart failure. Eligible trials will be those done in humans from 1 January 1990 to 30 November 2016 and reported in the English language. Nonrandomized trials and crossover studies will not be included in the survey. For purposes of the survey, adequate reporting of abstract will be determined by considering the proportion of studies reporting up to 16 items in the checklist. Using a 95% confidence interval approach, the number of required pilot trials abstracts (n) for the survey will be given by:

$$n=1.962(P_0(1-P_0)/E^2)$$

where 1.96 is the z-score associated with a 95% confidence interval, P₀ is the prior estimate of the proportion of studies with adequate reporting of abstract and E is the target margin of error for the estimate. With a margin of error of E = 0.10, our expectation is that the number of studies with an adequate report of abstracts will be calculated at P₀ = 0.60 [31], [32] (Table 2).

Table 2.

Estimation of sample size based on 95% Confidence Interval.

		P0
		0.50	0.55	0.60	0.65	0.70	0.75	0.80	0.85	0.90	0.95
E	0.01	9604	9507	9219	8739	8067	7203	6147	4898	3457	1824
	0.05	384	372	368	349	322	288	245	195	138	73
	0.10	96	95	92	87	80	72	61	49	34	18

P₀ = Prior estimate of studies with adequate reporting.

E = Margin of error for the estimate.

4.2. Statistical analyses

As shown in Table 3, the analysis will be divided into two sections.

Table 3.

Analysis plan.

Objectives	Outcome	Explanatory Notes	Hypothesis	Method of analysis
Primary: To review the quality of abstracts of pilot trials in heart failure in the past 26 years (1990–2016), using CONSORT extension for reporting of pilot trials	1Overall quality of the reporting of abstracts 2Quality of reporting of the individual items	Publications done from January 1990 to November 2016 1CONSORT endorsement by the journal 2Number of centers (single vs. Multi-centric) 3Type of intervention (pharmacological vs. Non-pharmacological) 4Significance of results for primary outcome (significant vs. Non-significant) 5Source of funding (industry funded vs. Non-industry funded	The quality of reporting of abstracts is suboptimal	1Presented as frequencies and percentages using descriptive analysis. Poisson regression analysis (or negative binomial regression) to present factors related to reporting at 95% C·I
Secondary: 1Identify aspect of checklist consistently reported in pilot trials, 2Identify factors associated with good reporting of abstracts of pilot trials				2Regression analysis

4.2.1. Primary outcome measures

Reporting quality by presenting reported and unreported items on the checklist as frequencies and percentages using descriptive analysis.

4.2.2. Secondary outcome measures

Negative binomial regression analysis will be used to determine the study characteristics associated with greater statistical reporting. Count of adequately reported items on the checklist will be the dependent variable. The independent variable will include the following characteristics linked to reporting quality in previous publications [11], [23], [24], [26], [27], [29]; journal impact factor, endorsement of CONSORT, sample size, single or multiple sites, type of study (pharmacological or non-pharmacological), abstract format (structured or unstructured). Incident risk ratio (IRR) will be calculated to evaluate factors associated with better reporting. The result will be presented as IRR with 95% confidence interval and associated p-values. The criterion for statistical significance will be set at alpha = 0.05. We will use SPSS software version 23 (Chicago, IL) for all analyses.

5. Ethics and dissemination

Formal ethical approval is not required for the proposed survey as data collection is based on publicly available reports. This protocol was written according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for Protocols (PRISMA-P) [33]. We will submit this systematic survey when completed to a peer-reviewed journal for publication, and the findings will also be presented at an upcoming conference.

6. Discussion

This planned systematic survey of quality of abstracts of pilot trials was registered prospectively in PROSPERO (CRD42016049911) and the protocol was written in line with PRISMA-P. It is to the best of our knowledge the first of such review using the CONSORT extension for reporting abstracts of pilot trials. The heart failure trials to be analyzed were published before the introduction of the CONSORT extension. However, the work will serve as the basis to evaluate improvement when the extension is fully in operation.

7. Authors' data sharing and contributions

All authors contributed to the protocol and approved the final manuscript. LT and BM were responsible for the conception of the survey. GI was involved in the search strategy. GI and LT designed the survey. GI, MZ, MC were involved in designing and testing of the data extraction form. GI wrote the initial draft, GI and LT contributed to improvements in the manuscript and BM and LT critically revised the final draft.

All authors approved the final written manuscript.

Responsibility for statistical analysis Plan: MC.

Guarantor of the systematic survey; Prof. Lehana Thabane.

8. Funding statement

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

9. Competing for interests' statement

The authors do not have any competing interests to report.

Appendix A.

Heart failure Pilot Trial Search Strategy.

PubMed search

Heart failure [Mesh] OR.

Heart failure OR paroxysmal dyspnea OR diastolic heart failure OR systolic heart failure OR Cardiac Failure OR Heart Decompensation OR Myocardial Failure OR Congestive Heart Failure OR Ventricular dysfunction OR cardiac insufficiency OR myocardial failure OR myocardial insufficiency.

AND.

Filters for RCT's in PubMed courtesy University of Cape Town Library

Randomized controlled trial [pt] OR controlled clinical trial [pt] OR randomized controlled trials [mh] OR random allocation [mh] OR double-blind method [mh] OR single-blind method [mh] OR clinical trial [pt] OR clinical trials [mh] OR (“clinical trial” [tw]) OR ((singl* [tw] OR doubl* [tw] OR trebl* [tw] OR tripl* [tw]) AND (mask* [tw] OR blind* [tw])) OR (placebos [mh] OR placebo* [tw] OR random* [tw] OR research design [mh:noexp] OR (comparative study) OR (comparative studies) OR (evaluation studies) OR (evaluation study) OR follow-up studies [mh] OR prospective studies [mh] OR controlled [tw]OR controls [tw]OR control [tw] OR prospectiv* [tw] OR volunteer* [tw]) NOT (animals [mh] NOT human [mh])

AND.

Pilot projects [MeSH] OR Feasibility Studies [MeSH] OR pilot OR feasibility.

Filter by years.

Filter by years (1 January 1990–1 May 2016)

Africa-Wide (Ebsco) and Web of Science

Heart failure OR paroxysmal dyspnea OR diastolic heart failure OR systolic heart failure OR Cardiac Failure OR Heart Decompensation OR Myocardial Failure OR Congestive Heart Failure OR Ventricular dysfunction OR cardiac insufficiency OR myocardial failure OR myocardial insufficiency.

AND.

Clinical trial OR randomized controlled trial OR randomized controlled trial OR random allocation OR double-blind OR single-blind OR placebo OR random research OR comparative study OR evaluation study OR follow up OR follow-up OR prospective OR control OR volunteer OR single mask OR double mask OR treble mask OR triple mask OR single blind OR double-blind OR treble blind OR triple blind.

AND.

Pilot OR feasibility.

Filter by years (1 January 1990–1 May 2016)

Scopus-

Heart failure OR paroxysmal dyspnea OR diastolic heart failure OR systolic heart failure OR Cardiac Failure OR Heart Decompensation OR Myocardial Failure OR Congestive Heart Failure OR Ventricular dysfunction OR cardiac insufficiency OR myocardial failure OR.

Myocardial insufficiency.

AND

Clinical trial OR randomized controlled trial OR randomized controlled trial OR random allocation OR double-blind OR single-blind OR placebo OR random research OR comparative study OR evaluation study OR follow up OR follow-up OR prospective OR.

Control* OR volunteer OR single mask OR double mask OR treble mask OR triple mask OR.

Single-blind OR double-blind OR treble blind OR triple blind AND pilot OR feasibility.

Appendix B. CONSORT extension for reporting abstracts of pilot trials ⁹

Item	Extension for pilot trials	Judgment		Score
		Yes	No
Title	Identification of study as a randomized pilot trial
Trial design	Description of pilot trial design (e.g. parallel, cluster
METHODS
Participants	Eligibility criteria for participants and the settings where the pilot trial was conducted
Interventions	Interventions intended for each group
Objective	Specific objectives of the pilot trial
Outcome	Pre-specified assessment or measurement to address the pilot trial objective(s)1
Randomization	How participants were allocated to the interventions
Blinding (masking)	Whether or not participants, caregivers, and those assessing the outcomes were blinded to group assignment
RESULTS
Numbers randomized	Number of participants screened and randomized to each group for the pilot trial objective(s)1
Recruitment	Trial status2
Numbers analyzed	Number of participants analyzed in each group for the pilot objective(s)1
Outcome	Results for the pilot objective (s); including any expressions of uncertainty1
Harms	Important adverse events or side-effects
Conclusion	General interpretation of the results of pilot trial and their implications for the future definitive trial
Trial registration	Registration number for pilot trial and name of trial register
Funding	Source of funding for pilot trial

¹Space permitting, list all pilot trial objectives and give the results for each. Otherwise, report those, which are a priori, agreed as the most important (main) to the decision to proceed with the future definitive trial.

²For conference abstracts.

Appendix A. Supplementary data

The following is the supplementary data related to this article: