Oral presentation bias: a retrospective cohort study

Abstract

The aim of this paper was to assess oral presentation bias at a national level. This was a retrospective cohort study with initial characteristics of the approved protocols extracted from the committee's archives, and follow‐up characteristics obtained from a questionnaire mailed to the principal investigators. A representative sample of French research ethics committees (25/48), the only committees legally endorsed for ethical authorisation in biomedical research, were studied. All completed research protocols, which had been approved in 1994 by these committees, were included. Initial characteristics (design, study size, investigator) of completed studies and follow‐up information (direction of results, rates of publication and rates of oral presentation) were collected. Complete information on results and their dissemination was available for 248 completed non‐confidential protocols. Half of these (49%) were declared as orally presented. The observed ranking for strategies to disseminate results was the following: orally presented and published, published only, neither orally presented nor published and orally presented only. Confirmatory results were more often orally presented, with an adjusted OR of 6.4 (95% CI 2.69 to 15.22). Other associated variables are the following: national/international scope of the study, protocol writer's university status, adverse events and interim analysis. There is a trend to submit or accept confirmatory results for oral presentations: meetings are a biased representation of research, and oral presentation bias could even be higher than publication bias.

Publication bias is usually defined as the tendency on the part of authors, journals and sponsors to submit and publish more studies with significant results.¹ Although scientific publication is considered as the main mode of disseminating results, it is not unique, as oral presentations in meetings are also a frequent and sometimes unique means of dissemination and are perhaps more specific and up‐to‐date for clinicians. It is therefore of great importance to know whether results presented orally are representative of the research conducted. It has been shown that investigators were not very keen on writing and submitting their results for publication when these results did not provide what was considered to be relevant information.²,³,⁴,⁵,⁶ The same configuration probably occurs with regard to submission for oral presentation in scientific meetings, but this has never been studied. As a parallel, space is limited in journals and time is limited in meetings. Scientific boards, like editorial boards, have to make choices and might also select “confirmatory results”. As journal reviewers are influenced by results,⁷ it has also already been shown that abstracts with significant results were more easily accepted for oral presentation.⁸ A review of abstracts and a poster from a meeting of the medical professional society showed that 87% of studies reported positive results.⁹

As a consequence, oral presentations in meetings might be a biased representation of the research conducted, which could be called “oral presentation bias”.

To our knowledge, no attempt has been made to assess this “oral presentation bias”, defined as the tendency, on the parts of researchers and reviewers of scientific boards of meetings, to submit or accept significant results for oral presentation. We therefore evaluated this potential bias on data originally collected to evaluate publication bias.²

Methods

French research ethics committees (RECs) are defined by law, and their membership is consistent across the country. Moreover, investigators can only submit their protocols to committees from the same administrative area, ensuring that all committees have more or less the same type of protocols to review.

We surveyed an exhaustive sample of protocols extracted from a sample of 25/48 (54%) RECs. Committees were randomly chosen to ensure a geographical cross section representative of the French administrative areas (the number of committees in each area depends on population size).

The outcome was assessed for every approved protocol: were completed studies presented orally? Our main hypothesis was that studies with confirmatory results were more likely to be orally presented than those with invalidating or inconclusive results.

Subjects

We included all clinical protocols newly approved between 1 January 1994 and 31 December 1994 by any of the 25 participating French committees. The year 1994 was chosen to ensure that most protocols were completed by the time of data collection. We identified protocols completed with the following characteristics: not confidential, with a hypothesis tested, with available information on direction of results and known modalities of results' dissemination. In the other cases, oral presentation bias could not be assessed.

Definitions

The scope of the study was defined as national‐monocentric, national‐multicentric or international‐multicentric. Investigators had to classify their results globally according to the following: no hypothesis tested, results confirming study hypothesis (confirmatory results), results invalidating study hypothesis (invalidating results), or results not confirming or invalidating study hypothesis (inconclusive results).

We classified as “confidential” protocols describing research that the investigator reported was not intended to be disseminated.

Investigators were asked if their results were disseminated through “oral presentation in a meeting”; it was not specified whether the oral presentation was peer reviewed or if it occurred in a regional, national or an international setting.

Oral presentation was the only end point of interest, as abstracts were considered both a means for submission of oral presentation and a consequence (publication in conference proceedings).

Data collection

Protocol characteristics were collected from the RECs' files and follow‐up data from questionnaires mailed to the principal investigator.

To ensure standardisation of methods across sites, at each committee, research assistants, blinded to our study hypothesis, attended a formal training session on completion of questionnaires and abstraction of study characteristics. Completed forms were sent to the coordinating centre, where an identification number was assigned to each protocol to ensure investigator anonymity and confidentiality.

Research assistants were also locally responsible for obtaining follow‐up data from the principal investigator of each protocol using a mailed questionnaire. In the case of non‐response, principal investigators were locally contacted up to six times by mail or telephone. When no answer was obtained from the investigator, the local committee sent the questionnaire to the sponsor for completion in summer 2002.

Ethical considerations

This study was conducted according to the French law on epidemiological and descriptive studies. Data on each investigator were collected anonymously, no consent was required as no individual information was retrieved and no ethics committee approval was required. For research confidentiality, investigators of this study were blinded and were not aware of researchers' names.

Statistical methods

Statistical analysis was performed using SAS software version 9.1 by the Clinical Epidemiology Unit of the Hospices Civils de Lyon, Lyon, France.

Frequency distributions were obtained for all categorical and continuous variables (means, percentages and 95% confidence intervals (CIs)). Associations were considered significant when p values were <0.05.

Oral presentation bias was assessed with a logistic regression.¹⁰ To adjust for potential confounding factors, we introduced variables significant at the 0.25 level (in univariate analysis) in a forward stepwise logistic regression (p value for entry = 0.25; p value for stay = 0.15). The internal validity was assessed with the Hosmer–Lemeshow test.¹⁰

Results

Among the 501 completed studies, we excluded 181 (36%) protocols deemed confidential (fig 1). Protocols were also excluded when no hypothesis was tested (n = 32, 6.4%), and when the direction (n = 28, 5.6%) or dissemination (n = 12, 2.4%) of results were not known or not provided. A total of 248 protocols were available for our analysis of oral presentation bias, and table 1 presents their main characteristics.

Figure 1 Protocols included and dissemination of results.

Table 1 Characteristics of the 248 protocols.

	n (%)
Study topic
Drug testing	160 (65)
Phase II	53 (33)
Phase III	76 (47)
Phase IV	31 (20)
Cosmetics, nutrition	9 (4)
Medical device testing	18 (7)
Surgical and diagnostic procedures	14 (6)
Physiology	24 (10)
Other	23 (9)
Design
Descriptive or observational	36 (15)
Experimental	212 (85)
Non‐randomised	72 (34)
Randomised, no blinding	34 (16)
Randomised, single blinding	21 (10)
Randomised, double blinding	85 (40)
Funding
No funding	21 (8)
Private funding	167 (67)
Public funding	39 (16)
Mixed funding	21 (8)
Scope
National	182 (73)
Single centre	100 (55)
Multicentre	82 (45)
International multicentre	66 (27)

Dissemination of results

Results were disseminated through an oral presentation for 120 (49%) studies, and through a scientific article for 146 (59%) studies. Among studies orally presented, 90% (n = 108) had confirmatory results versus 76% for the overall sample (fig 1).

When results were inconclusive or invalidating, the hierarchy of dissemination strategies adopted by the investigator was the following: neither orally presented nor published, published, orally presented, and both orally presented and published.

Oral presentation bias

We found evidence of oral presentation bias, with a crude odds ratio (OR) of 5.25 (95% confidence interval (CI) 2.39 to 11.54) for confirmatory versus inconclusive results and 0.9 (95% CI 0.21 to 3.84) for invalidating versus inconclusive results. After forward stepwise regression, another four variables were included in the model: study scope (national/international), writer status of the study protocol, occurrence of adverse events and interim analysis (table 2). The direction of results remained highly significant, with an adjusted OR of 6.4 (95% CI 2.69 to 15.22) for confirmatory results versus inconclusive results.

Table 2 Variables statistically linked with oral presentation (multivariate analysis*).

	Oral presentation n (%)	OR (95% CI)
Scope
National	83/182 (46)	1
Multinational	37/66 (56)	2.61 (1.24 to 5.47)
Protocol's writer status
Sponsor	20/59 (34)	1
Sponsor and investigator	42/94 (45)	2.35 (1.07 to 5.2)
Investigator	58/95 (61)	3.88 (1.67 to 9.01)
Adverse events
None	100/194 (52)	1
Adverse events observed	19/54 (35)	0.43 (0.21 to 0.92)
Interim analysis
No interim analysis	85/203 (42)	1
Interim analysis	35/45 (78)	5 (2.09 to 11.93)
Results
Inconclusive results	9/44 (20)	1
Invalidating results	3/16 (19)	1.07 (0.23 to 5)
Confirmatory results	108/188 (57)	6.4 (2.69 to 15.22)

*Adjusted by using forward stepwise regression on the subset of variables linked at the 0.25 level in univariate analysis (other variables tested: funding, design, phase and study product); OR of 1 assigned to reference category.

When restricting the analysis to the unpublished studies (n = 102), the direction of results was also predictive of oral presentation (p<0.001).

Simultaneity of publication bias and oral presentation bias

The crude OR for results' dissemination (either oral presentation or scientific article) was 9.42 (95% CI 4.52 to 19.62) for confirmatory versus inconclusive results and 0.48 (95% CI 0.13 to 1.73) for invalidating versus inconclusive results.

Only 25% (63/248) of studies were neither published nor presented. Results were confirmatory for 40% (n = 25) of these studies, whereas the results of 88% (n = 163) of the studies with at least one presentation or one publication were confirmatory (fig 1).

Dissemination of results for descriptive‐only and confidential protocols

When no hypothesis was tested, publication was frequent (60%); on the contrary, when the investigator did not provide or know the direction of results, scientific articles were published in 14% of cases. Confidential studies were disseminated, oral or written, in only 15% of cases (data not shown).

Discussion

Half of the completed studies were orally presented 7 years later. Confirmatory results are 6.4 times more likely to be orally presented than non‐confirmatory results (OR 6.4, 95% CI 2.69 to 15.22).

Our study is the first to formalise the existence of oral presentation bias and to evaluate it. The term oral presentation bias itself has never been used so far, and previous observational studies on publication bias²,³,⁴,⁵,⁶ have only provided some rates of oral presentations.

Although this study was not only designed to investigate oral presentation bias, the data collected were structured to provide quantification. As oral presentation was not the main end point of our study, specific data such as scope of the meeting (regional, national and international), subject of the meeting (specialty meeting, consensus conference and annual meeting of a scientific association), sponsor of the meeting (hospital, pharmaceutical firms, associations, etc), existence of a peer‐review process, number of oral presentations per protocol (submission or not and reasons for, rejection or not), etc were not collected.

As we only asked whether the results were orally presented, a “yes” answer might cover, for example, oral presentation to local meetings, that is to say not peer‐reviewed scientific meetings. But we think that such informal meetings would take less time to prepare and that investigators would more easily present negative results; therefore, this would tend to confirm the null hypothesis and in this type of situation, the effect size of this bias would even be greater than that reported in our study.

What is already known

• To our knowledge, no attempt has been made to assess “oral presentation bias”.

• Only four observational studies have shown evidence of publication bias in biomedical research approved by research ethics committees.

What this paper adds

• There is also an “oral presentation bias”, different from publication bias.

• The tendency on the parts of researchers and reviewers of scientific boards of meetings to submit or to accept a presentation is 6.4 times greater for confirmatory results.

• Scientific meetings, as much as published literature, could globally be a biased representation of research as studies with confirmatory results are also more easily submitted and accepted.

We also restricted the oral presentation bias analysis to completed protocols, not confidential, with a hypothesis tested, and with available information on the direction of results and dissemination of results. These criteria were necessary to perform the analysis. However, the criterion of confidentiality could be discussed. We are convinced that all research without any exception should be published, but some protocols such as phase I studies (mainly deemed confidential) are usually not published or presented, and remain confidential. We decided to present them separately and verified that few were disseminated. However, these confidential studies did not differ markedly on the direction of results, and their exclusion did not produce any bias.

The major limitation of our study may be memory bias. Only 49% of completed, non‐confidential and non‐phase I studies were orally presented. We found this figure to be rather low, but it might be explained because investigators easily remember scientific publications, which are more prestigious, compared with oral presentations, which may be more commonplace with less of an effect on scientific promotion. Another hypothesis to explain the low figure of oral presentation is that meeting fees and travel expenses make an oral presentation more expensive than submitting a scientific paper.

Another limitation is that direction of results was provided by the investigator and that interpretations might be subjective.

A surprising result is that the rate of “published only” is greater than that of “orally presented only” and that this order remained the same for invalidating or inconclusive results. One might have thought that oral presentations would be more frequent than scientific publication, as found by Easterbrook et al,³ because the time required for preparing an abstract might be shorter than for an article and because gaining acceptance might easier for oral presentation than for scientific articles. This might be due to specificities of the French hospital and academic system,¹¹ yet to be identified if any.

Moreover, in the publication bias literature, another type of bias related to statistical significance was recently reported in two studies¹²,¹³: investigators were more likely to report statistically significant outcomes and failed to report others (outcome reporting bias). This bias might also exist in meetings.

In another recent paper,¹⁴ the authors surveyed 26 studies on publication bias. This review included studies on both publication bias based on protocols approved by a research ethics committees and on publication bias based on abstracts of meetings. These two publication biases should not be merged, as we have shown that abstracts presented are not representative of research. As a consequence, publication bias measured from abstracts of meetings (abstract‐based publication bias) should be differentiated from publication bias measured from research. Scientific meetings, often attended by doctors to keep knowledge or practices up‐to‐date, are globally a biased representation of research as oral presentation bias exists: researchers tend to submit studies with confirmatory results, which are also more easily accepted by scientific boards, as shown previously.