Reliability of disclosure forms of authors' contributions

Abstract

Background

The contribution disclosure forms used by medical journals to assess and confirm authorship are surveys of self-reported behaviour that follow the cognitive rules of psychometric instruments. We sought to analyze how autobiographical memory, defined as memory for events and issues related to oneself, affected the reliability of contribution forms for the judging of authorship of research articles.

Methods

We conducted a prospective study, which ultimately included 919 authors of 201 articles submitted to a general medical journal from July 2001 through December 2002. A authorship disclosure form with a checklist of 11 possible contribution choices for all authors was sent first to each article's corresponding author, who was asked to fill it out for all authors. A blank form was then sent to each author individually to disclose his or her own contribution to that article. The main outcome measure was test– retest differences between the corresponding authors' self-declarations, expressed in percent as the gross difference rate (GDR) for each article.

Results

More than two-thirds of the corresponding authors (69.7%) differed in at least 1 contribution choice between the 2 disclosure statements made about their own contributions. The reliability of their answers was low to moderate (GDRs > 10%), especially for contributions on the provision of study materials or patients or final approval of the article (GDR 22.9%), guarantor of the study (GDR 20.9%) and drafting of the manuscript (GDR 20.4%). As a proxy for their coauthors' contributions, corresponding authors also differed from them in the perception of noncorresponding authors' contributions, disagreeing in 69.4% of cases. Of the 718 noncorresponding authors, 204 (28.4%) met all the criteria for authorship set out by the International Committee of Medical Journal Editors according to the statement given by the corresponding author. When they described their own contributions, this prevalence increased to 40.5%.

Interpretation

Psychological factors such as autobiographical memory may confound contribution disclosures as an evaluation tool for authorship on scientific articles and affect responsible authorship and publication practices.

Authorship of scientific articles is the main measure of research productivity and an important requirement for academic advancement.¹ It is nonetheless burdened with problems, such as misconceptions about ownership of research² and misuse of author's responsibilities.^3,4 To address undeserved claims of authorship and increase the transparency of scientific publishing, many journals have introduced the practice of contribution disclosure, whereby authors are required to fill out a contribution and authorship statement form.⁵ Contribution disclosure forms for assessing authorship are also surveys of self-reported behaviour and thus follow the same cognitive rules as other questionnaires. When answering a question, a participant in a survey performs 5 cognitive tasks:⁵ understanding the question, recalling relevant behaviour, inferring and estimating the behaviour in question, mapping the answer to the response format, and “editing” the answer for social desirability.

In a previous study,⁶ we showed that the introduction of contribution disclosure practices in 3 major medical journals was paralleled by differences in the prevalence of authors who did not satisfy the standardized authorship criteria of the International Committee of Medical Journal Editors (ICMJE).⁷ In a randomized study⁸ published in the Croatian Medical Journal, a general medical journal, we demonstrated as well that the structure of the contribution disclosure form influenced the number of contributions reported by authors of submitted manuscripts and their compliance with the ICMJE authorship criteria. This confirmed that the cognitive task of mapping the answer to the response format of the contribution disclosure form influences the answers on the forms and therefore the attribution of authorship as defined by ICMJE.

To test how the construct of autobiographic memory, defined as a personal representation of general or specific events and personal facts,⁹ affects the reproducibility of statements about contributions to a submitted article indicated on forms that self-report past behaviour, we carried out a study in which we asked the corresponding authors to report their contributions to the same article on 2 different occasions: first on the contributions of all coauthors (i.e., acting as a proxy on their behalf) and later, when all authors were sent the same (blank) form, as an individual asked to declare his or her own contribution as a coauthor.

Methods

All manuscripts submitted to the Croatian Medical Journal, an international general medical journal published in Croatia,^8,10 were considered eligible for the study. All articles submitted from July 2001 through December 2002 were considered (n = 318), although those without multiple authors (12.3%) were excluded (Fig. 1). The final sample comprised 201 research articles with 2 or more authors and a total of 919 coauthors, with a median number of coauthors per article of 4 (95% confidence interval [CI] 4–5; Table 1). Upon receipt of each submitted article, a number of authorship forms equal to the number of authors of the manuscript was sent to the corresponding author. Each form (Appendix 1, available at www.cmaj.ca/cgi/content/full/176/1/41/DC1) presented 11 possible contribution choices for declaration of authorship. Four to 6 weeks later, after these “group” forms were returned by the corresponding authors, the same (blank) form was sent to each coauthor (including the corresponding author) individually. To increase the return rate of the authorship forms, no authors were informed of any editorial decisions about their manuscript before the return of all forms sent in the second survey.

Fig. 1: Flowchart showing the study exclusions that led to the final sample size.

Table 1

All correspondence with authors was in English, the official language of the journal. The authorship statement form had a general statement on editorial research that did not disclose the study aim or details but asked for consent to participate in general journalistic research (see the supplementary material in section 5 of the form). The Ethics Committee of the Osijek University School of Medicine gave ethical approval for the study.

The authorship form (Appendix 1, at www.cmaj.ca/cgi/content/full/176/1/41/DC1) had an introductory paragraph describing the authorship criteria of ICMJE⁷ and emphasized that the Croatian Medical Journal follows these criteria. The list of 11 contribution choices were ordered randomly with use of a random number generator, as follows: (1) participation in the conception and design of the study; (2) analysis and interpretation of data; (3) provision of study material or patients; (4) collection, assembly and possession of the raw data; (5) statistical expertise; (6) drafting of the article; (7) critical revision of the manuscript for important intellectual content; (8) final approval of the article for publication; (9) obtaining funding for the study; (10) administrative, technical or logistical support; and (11) willingness to guarantee the integrity of the entire study.

The 1st, 2nd, 4th or 5th contribution choices constituted eligibility for authorship in accordance with the first ICMJE criterion (conception and design of the study or data acquisition or analysis and interpretation) of the Uniform Requirements for Manuscripts Submitted to Biomedical Journals;⁷ the 6th or 7th, in accordance with the second criterion (manuscript drafting or revision); and the 8th, with the third criterion (final approval). The authorship of anyone whose contributions did not meet all 3 ICMJE criteria (i.e., those who disclosed only the 3rd, 9th, 10th or 11th choices) was deemed to be honorary. Satisfaction of ICMJE authorship criteria was evaluated for all authors according to statements returned in either survey.

We measured test–retest differences in the contribution choices indicated by corresponding authors as the gross difference rate (GDR)¹¹ for that article. The differences between individual coauthors' declarations and those made on their behalf by the corresponding authors were expressed as differences in proportions of group and individual forms.

Differences in proportions of variables between the group and individual forms were tested with the exact McNemar test. Differences in the total number of authors' contributions stated on group and individual forms were tested with the Wilcoxon signed-rank test (S). Differences in the median number of byline authors between subgroups of articles (those with and without honorary authors) were tested with the Mann–Whitney U test. GDR was used to assess the reliability of answers given by corresponding authors about their own contributions.¹¹ It was calculated as GDR = 100 × (b + c)/n, where (b + c) represents the sum of the off-diagonal frequencies in the cross-tabulation table of the test–retest answers. Reliability was considered high (GDR < 10%), moderate (GDR 10%–20%, boundaries included) or low (GDR > 20%).¹¹ All p values were 2-tailed; a Bonferroni correction was used for subgroup comparisons.

Results

When we compared the contribution choices made at 2 different times by the corresponding authors on the same type of form for the same article, we found that more than two-thirds (140/201, 69.7%) differed in at least one contribution choice. The most consistent choice was “conception and design of the study” (Table 2): only 8 of 201 test–retest pairs differed in the declaration on the category, considered highly reliable (GDR 4.0%, 95% CI 2.0%–7.7%). The reliability of other contribution choices was medium to low. Among the choices eligible for ICMJE authorship criteria, the lowest reliability was observed for the category “final approval of the article,” followed by “drafting of the article” and “critical revision of the manuscript for important intellectual content” (Table 2). Interestingly, corresponding authors were unsure about their own contributions to the provision of study materials or patients, which had the lowest reliability (GDR 22.9%, 95% CI 17.6%–29.2%).

Table 2

Declaration choices made by the corresponding authors for their coauthors also differed from those made by the coauthors for themselves (Table 2). Coauthors chose more contribution categories on individual authorship forms than corresponding authors did for them on group forms (S = 42578.5; p < 0.001; Table 3). Corresponding author and coauthors also disagreed on at least one choice for coauthor contributions in 498 of 718 cases (69.4%), although they agreed consistently on 3 contribution choices: “drafting of the article,” “obtaining funding” and “administrative, technical or logistical support” (Table 2). “Final approval of the article” was the category with the largest observed difference in prevalence between the declarations provided by the corresponding author for all authors than those made individually by each author (difference 14.4%, 95% CI 10.9%–17.7%; p < 0.001).

Table 3

Although individual declarations reported a greater variety of contribution choices (i.e., more choices in total) than those filled out by corresponding authors on group forms (Table 1), the number of articles in which all coauthors satisfied all 3 ICMJE criteria for authorship did not increase significantly (difference 4%, 95% CI –2% to 10%; p = 0.24; Table 3). Neither was there any significant difference in the prevalence of honorary authors on articles from Croatia or other countries (range 71%–77%), regardless of who filled out the contribution form.

Of the 607 honorary authors defined according to the group forms, 161 (26.5%) met all ICMJE criteria when they filled out their individual form. Conversely, of 312 authors who met all ICMJE criteria according to group forms, 48 (15.4%) failed to meet them on their individual forms. In a total of 64 (31.8%) of 201 articles, all authors listed on the byline were honorary authors according to the statements given by corresponding authors. When authors themselves described their own contributions, this prevalence decreased to 20% (difference –12%, 95% CI –18% to –6%; p < 0.001). According to either statement, about 25% of all articles had no honorary authors listed in the byline (Table 3). Articles with honorary authors had significantly longer bylines then those without honorary authors according to either type of form (z = –2.837 for group forms, p = 0.005; z = –3.674 for individual forms, p < 0.001).

When we matched the declared contribution of corresponding authors on either form against the ICMJE criteria, we found that these criteria were satisfied to a greater extent according to the individual than group contribution form (difference 13%, 95% CI 6%–20%; p = 0.001; Table 3). Of the 718 noncorresponding authors, 28.4% fully met the ICMJE criteria for authorship according to the statements given by corresponding authors. When they themselves described their own contributions, this prevalence increased to 40.5% (difference 12.1%, 95% CI 8.8%–15.4%; p < 0.001).

Among honorary authors who missed 1 out of the 3 ICMJE criteria, it was the third (final approval of the article) for almost half of them, whether the information came from group or individual forms (Table 3). In combination with the other 2, this criterion was missed by 88.8% of the honorary authors on the group and 83.0% of honorary authors on their individual contribution form. Honorary authors mostly missed any 2 of the 3 ICMJE criteria; among those who missed all 3, none were corresponding authors.

Interpretation

Our study showed that the autobiographical memory construct, the memory for events and issues related to oneself,⁹ is involved in the cognitive aspect of filling out contribution declaration forms and confounds the conclusions about self-reported contributions toward the authorship of scientific articles, as well as their reliability. In psychometric terms, reliability is defined as the extent to which a test is dependable, stable and consistent when administered to the same people on different occasions.⁵ The existence of significant discrepancies between declarations made at 2 different times by corresponding authors about their own work on a single manuscript suggests that they had problems recalling relevant events from memory.

Research into retrieval of autobiographical memory shows that activities are the best retrieval cues, followed by location of and participants in the event.¹² Because the work on a manuscript to be submitted to an academic journal is an important part of being a scientist,¹³ such events should be well represented in memory and recalled easily. Moreover, the period between the 2 surveys in our study (4–6 weeks) should not have influenced the accuracy of memory, since both surveys were sent well after the stabilization of memory that occurs after its initial detrition.¹⁴ The fact that corresponding authors, presumably the people most knowledgeable about their manuscripts, did not consistently report on their own past behaviour may mean that the behaviours related to the article were poorly represented in memory or that they did not fill out the contribution disclosure forms truthfully, editing one or both for social desirability.⁵ Testing these possible explanations would require experimental settings not possible within the framework of editorial communication with authors. However, available theoretical and experimental data,⁵ coupled with the well-known pressure to publish among scientists,¹³ favour the second explanation of self-editing for social desirability.

Our study confirmed that reports about others' behaviours, such as proxy reports on the contributions of one's coauthors, were not an accurate way of assessing authorship. As has been shown by well-controlled experimental psychological observations,⁵ proxy reports about behaviours of others are burdened by a subjective inference process in estimating behaviour. Proxy observers base their reports largely on their assumptions about that person, rather than on their memory of the event. They may not be fully aware of others' behaviour, and unless it was extreme or memorable, this behaviour is usually not well represented in their memory.⁵

Our study is limited by its survey design, but our findings are strengthened by a very high response rate, since filling out authorship forms is mandatory for manuscript processing in most science journals. Generalizability of the findings to other medical journals and research and academic settings can also be questioned. However, the results were not likely influenced by the type or quality of the journal, because we primarily studied how people reported on their behaviour and not the practices in a specific journal. Research in this area of psychology has been performed in different settings and cultural environments, with consistent results.⁵ Respondents from our study came from different settings, and the findings did not differ among manuscripts from different geographical regions.

Psychometric analysis of contribution disclosure reports has indicated that they are an unreliable source of information. The present study, which addressed recalling relevant behaviour when answering a self-report, and our previous study showing that the form of asking questions shapes the contribution declaration⁸ provide evidence that the contribution declaration and the authorship criteria themselves have been introduced into the scientific-publication process without adequate evidence for all aspects of their validity.

As journal editors we need much more evidence to assume the right to judge or regulate¹⁵ matters of authorship, which primarily belongs to the wider academic and research communities.¹⁶ Although surveys show that researchers often do not agree with the ICMJE criteria or even know what they are,¹³ intervention studies have shown that a simple instruction given to medical students improves their knowledge and use of ICMJE authorship criteria.¹⁷ Contributorship statements can be prospectively drafted and revisited before manuscript submission to the satisfaction of all stakeholders in a research project.¹⁸ Offering a Likert-style quantification scale as a reference for author's contributions instead of the usual yes/no checkboxes might increase the number of contributors deserving authorship status.¹⁹ Until there is enough evidence to propose fair, universal and reliable guidelines for authorship, perhaps the editors should ask all authors a simple, open-ended question: “Why do you think you deserve to be the author of this manuscript?”