Studying Adherence to Reporting Standards in Kinesiology: A Post-publication Peer Review Brief Report

Abstract

To demonstrate how post-publication peer reviews—using journal article reporting standards—could improve the design and write-up of kinesiology research, the authors performed a post-publication peer review on one systematic literature review published in 2020. Two raters (1^st & 2^nd authors) critically appraised the case article between April and May 2021. The latest Journal Article Reporting Standards by the American Psychological Association relevant to the review were used: i.e., Table 1 (quantitative research standards) and Table 9 (research synthesis standards). A standard fully met was deemed satisfactory. Per Krippendorff’s alpha-coefficient, inter-rater agreement was moderate for Table 1 (k-alpha = .57, raw-agreement = 72.2%) and poor for Table 9 (k-alpha = .09, raw-agreement = 53.6%). A 100% consensus was reached on all discrepancies. Results suggest the case article’s Abstract, Methods, and Discussion sections required clarification or more detail. Per Table 9 standards, four sections were largely incomplete: i.e., Abstract (100%-incomplete), Introduction (66%-incomplete), Methods (75%-incomplete), and Discussion (66%-incomplete). Case article strengths included tabular summary of studies analyzed in the systematic review and a cautionary comment about the review’s generalizability. The article’s write-up gave detail to help the reader understand the scope of the study and decisions made by the authors. However, adequate detail was not provided to assess the credibility of all claims made in the article. This could affect readers’ ability to obtain critical and nuanced understanding of the article’s topics. The results of this critique should encourage (continuing) education on journal article reporting standards for diverse stakeholders (e.g., authors, reviewers).

INTRODUCTION

We sought to contribute further understanding on ways the write-up of articles published in kinesiology journals could be improved. Beyond using critical appraisal tools by manuscript authors and editors to assess the strengths and limitations of a manuscript (23), another method would be to engage in post-publication peer review of published work (13, 28). While traditional peer review generally improves the quality of the writing (35), a host of issues that call into question the credibility of scientific literature continuously slip past reviewers and editors or may be deemed non-pertinent due to publication constraints of a journal (e.g., page length, scope, available reviewers) (15, 34). The following are example issues that seem common to the published research literature: (a) missing a priori sample size calculation, (b) vague or incomplete write-up of systematic methods used in the study, (c) undisclosed change to a study’s analytic plan and (d) only reporting statistically significant results (28). Published research in kinesiology is not immune to these reoccurring issues (11).

Continuing educational programs focused on journal article reporting standards can aid both researchers and students in producing comprehensive and credible research. A call for more education in this area was also made in a 2021 presentation to the field, titled “A Content Analysis of Papers Published in Kinesiology Journals” (5). The authors studied 270 peer-reviewed research articles in kinesiology, and they found most lacked key detail in their reports (e.g., about their analytic plan, study instruments, or findings). Of note, many subdisciplines and journals in kinesiology require authors to compose their manuscripts in APA style (i.e., using the publication standards of the American Psychological Association [APA]). This adherence, however, may often entail a predominant focus on understanding and applying APA standards specific to layout and source citation. Perhaps unbeknownst to many consumers and producers of research, the APA has commissioned the creation of reporting standards for an array of study designs, which were disseminated vis-à-vis multiple editions of the Association’s official publication manual (i.e., 2010, 2020; 1–2).

The purpose of this study was to demonstrate how post-publication peer reviews—using journal article reporting standards—could improve the design and write-up of kinesiology research. Previous literature reviews have focused on the utility of published research to researchers and other professionals (11, 34). Thus, our review study focused on the ways peer-reviewed research in kinesiology may compare to APA journal article reporting standards specifically. At the time of our study, we found no published research which analyzed the degree that kinesiology research adhered to journal article reporting standards by the APA, which is a widely endorsed professional association in the field. Thus, the findings of this post-publication peer review should promote awareness of the APA journal article reporting standards. The focus of our critique aligns with aims to provide space for discussing ways to improve the science of kinesiology and transparency within its published scholarship (11).

METHODS

This research was carried out fully in accordance with the ethical standards of the International Journal of Exercise Science (21). Given this study did not involve human subjects, review and pre-approval of its protocol by the authors’ institutional review board was not required.

The case article used in this post-publication peer review was a 2020 publication of a systematic review study by Buja and colleagues, titled “Health Literacy and Physical Activity: A Systematic Review” (10). We first located the case article as a reference to a peer-reviewed presentation we co-authored, and which the second author (JDT) delivered in January of 2021 (32). Following the presentation, the second author invited the first author (NMW) to critique the article by Buja and colleagues as an independent study project. The first author was a senior-level undergraduate student at the time, and the objective was to deepen their understanding of principles for thinking critically about published research. Additionally, the goal of the project was to provide the first author additional experience engaging in discourse in kinesiology and disseminating research in professional forums. The project period of the present study was April 2021 to May 2021. A critical appraisal of the select peer-reviewed article (i.e., a single artifact case study) was performed, using a mixed-methods form of content analysis (31). Specifically, methodology for qualitative and quantitative inquiry were used in the present study (29).

According to Appelbaum and colleagues within their 2018 article reporting on an update to the APA journal article reporting standards, the standards were revised to expand standards for quantitative research reports and to establish new reporting standards for qualitative research (3). In their discussion of the standards, Applebaum and colleagues (3) stated that it would be beneficial for anyone conducting research in the social sciences to “acquire the habit of utilizing reporting standards as a part of their formulation of how scholarly research is reported.” The positive impact that using journal article reporting standards could have on the quality of research reports extends to any scientific discipline involving human participants or focused on dimensions of human ecology. Specifically, widespread utilization of reporting standards would lend increased credibility to quantitative and qualitative research conducted in kinesiology and could enhance the accuracy of replication studies in the field. Before performing our critical appraisal, we made the a priori decision to perform a constructive post-publication peer review of the case article, focusing on its reporting strengths and limitations (or areas for improvement; for further discussion, see 12, 17).

Protocol

The reference article by Appelbaum et al presented the latest APA journal article reporting standards at the time of the present study (3); their article was used to construct a coding form to critically appraise the case article used in this post-publication peer review study. Specifically, two sets of APA journal article reporting standards were used: Table 1 focused on general standards for quantitative research and Table 9 focused on inclusive standards for any research synthesis study. The coding protocol required a conservative approach (i.e., an “all or nothing” decision-rule or heuristic evaluation, 31). For a journal article reporting standard to have been deemed “satisfied,” all criteria to that standard needed to have been observable in the article’s written text (or referred to within its supplemental files). If the composed text of a section (e.g., Methods section) did not meet one or more criteria of a journal article reporting standard (i.e., partial or no match), then the entire section was deemed “unsatisfactory” for that standard (e.g., for Table 1, quantitative research reports: the standard to report data diagnostics contains five criteria, the standard to report analytic strategy contains three criteria; 3, p. 7). If a code of “unsatisfactory” was reached, reasoning for the decision was provided. This approach to coding the article was employed to support good inter-rater reliability by standardizing the coding procedure (30). Coding was performed by two reviewers (1^st & 2^nd authors). Before independent coding commenced, the authors discussed the coding protocol and reviewed the text of the published APA journal article reporting standards published within the article by Appelbaum and colleagues (3). This pre-coding meeting was to ensure that the authors had the same understanding for how to implement the coding protocol before they independently coded the case article. Figure 1 illustrates the integration of both Table standards into a single electronic coding form (a spreadsheet file). Prior to this article’s submission for publication consideration, its protocol, analytic plan and results sections were independently reviewed and deemed veracious by one university faculty member from another institution, with expertise in quantitative research methods and writing transparent and replicable research reports.

Table 1.

Inter Rater Reliability Results.

APA Journal Article Reporting Standards	Raw Percent (%) Agreement	Krippendorf’s Alpha	Agrees	Disagrees	Level of Agreement
Table 1	72.2	0.57	26	10	Moderate
Table 9	53.6	0.09	15	13	Poor

Note: Rater reliability for the coding form was checked, and a 100% consensus on each discrepancy was reached after discussion. The values for raw percent agreement, frequency of agrees and disagrees, and Krippendorff’s alpha were computed using a valid and reliable webtool (14). The interpretive cut-points by Landis & Koch were used (19).

Figure 1.

The novel integrated coding form developed for the study and used to code the case article.

Before discussing discrepancies in coding following each author’s independent review, we made an amendment to the integrated coding form that allowed for a third code signifying “not applicable to the article”. The post-hoc addition of this category was built upon the independent inclinations of each reviewer that it was necessary in their respective analyses. Inter-rater reliability was assessed for each table independently. Table 1 presents both the raw percent agreement between raters and the statistic coefficient for inter-rater reliability (Krippendorff’s alpha), both of which were computed using a webtool shown to be valid and reliable (14). APA journal article reporting standards Table 1 had a moderate level of agreement that was nearly substantial, and APA journal article reporting standards Table 9 had a poor level of agreement (19). Following independent coding, all discrepancies were discussed and a 100% agreement on each discrepancy was reached.

Statistical Analysis

Descriptive statistics were used to identify trends for qualitative evaluation of the strengths and weaknesses of the case study article in terms of its write-up (31). Descriptive trends were interpreted and evaluated using the agreed upon notations made about the case article within the study coding form (31). The percentage of standards met was computed for each section of the case article and overall (i.e., count for satisfactory divided by count for unsatisfactory, multiplied by 100). Standards deemed “Not Applicable” were excluded from the computation (i.e., subtracted from the count for the denominator).

RESULTS

There were nine standards deemed as not applicable, all from Table 1 (see Supplemental File 1). Thus, the total number of standards analyzed in the present study was 54 out of 63 (n = 26 from Table 1, n = 28 from Table 9). According to both tables, the Abstract, Methods, and Discussion sections of the case article generally needed more clarification or detail to fully meet the APA reporting standards. When evaluating reporting standards for a research synthesis, the following sections were predominantly incomplete: Abstract (100% incomplete), Introduction (66% incomplete), Methods (75% incomplete), and Discussion (66% incomplete). Figure 2 displays the percent of standards met across each main section of the case article. When combining both tables, the article fully met 27 of 54 reporting standards (50%). A higher number of reporting standards were met for Table 1 than Table 9 across each article section, except for the title section where a tie was observed.

Figure 2.

Percent of journal article reporting standards met for each section of the case article.

The results of our post-publication peer review showed that there was sufficient detail to help the reader understand both the scope and the logic behind the study and the author’s decisions within the case article. However, critical information was missing from the case article that would support the replicability of the study. For example, readers were not provided with sufficient detail for assessing the credibility of each claim made by the authors (e.g., an explication or citation). Table 2 of this report provides an illustrative example of the reporting limitations identified within the case article. Table 3 of this report provides an illustrative example of the reporting strengths identified within the case article. Supplemental File 1 for this report presents the full results of each standard met and not met for the journal article reporting standards in Table 1 (quantitative research standards). Supplemental File 2 for this report presents the full results of each standard met and not met for journal article reporting standards in Table 9 (research synthesis standards).

Table 2.

Limitations.

Article Section	Synthesized Critique Across Raters
Article title	The article title did not address all criteria of the standard (e.g., indicate the theoretical issue investigated, the population studied).
Abstract	Objectives. The authors did not state a hypothesis tested in their study or problem/question investigated vis-à-vis their systematic review. Specifically, the precise issue, question, or problem investigated is presented in the Introduction section of the article but not in the Abstract. Conclusion. While a systematic and critical appraisal of studies included in the review concerning strengths, limitations, and bias risk were performed by the authors using standardized forms and reported within Supplemental Content files, key findings of that specific methodology were not presented in the Abstract (nor the Results section of the main text).
Discussion	The Discussion section at times lacked three key elements necessary for judging credibility of claims: overall quality appraisal of evidence reviewed, provide alternative explanations to trends observed, and highlighting how results of the present review were similar to and different from previous reviews.

Table 3.

Strengths.

Article Section	Synthesized Critique Across Raters
Abstract	The Abstract clearly stated the study design for the review study. Of note, demographic exclusion criteria stated within the article’s Method section were not listed within the Abstract, leaving the reader to assume no demographic restrictions were applied.
Results: Figure 1 (p. 1261)	The Authors provided a clear and concise flowchart, depicting the process that led to the study sample. A noteworthy strength was the inclusion of a breakdown of the number of materials across the exclusion criteria.
Results: Table 1 (p. 1262–1270)	Authors presented results of each study in a comprehensive table that aided in facilitating understanding of the scope of each study reviewed.
Discussion	The authors provided sufficient information to aid future research and policy development. The authors state that use of education-focused interventions is encouraged; given the findings of their systematic review, they encouraged public policy that focuses on a population-level health literacy promotion. Concerning implications, the authors acknowledged diverse measurement tools may lead to dissimilar results (this claim was substantiated with appropriate reference material).

DISCUSSION

A growing view on how to ensure rigorous research is conducted and reported properly is through post-publication peer review (13, 28). Given post-publication peer review is in the public domain and focuses on critiquing already published research, it is a highly sensitive undertaking. The aim of this brief report was to demonstrate a way to ensure post-publication peer reviews are constructive. We believed using reporting standards endorsed by stakeholders within the scientific community would meet this aim. Reporting standards endorsed (or adopted) by reputable professional societies are based on principles for sound and pragmatic methodology (3). That is, they are theoretically grounded, and they support replicable and reproducible research results (11, 28). Moreover, they have been empirically verified to mitigate against biased research practices or assumptions known to lead to flawed results or flawed interpretations of results (13, 26). We critically appraised one secondary research article published in the kinesiology and health promotion literature following a traditional peer review process, using an integrated coding form based upon endorsed reporting standards for scholarly work. The results of our case study demonstrate that the use of journal article reporting standards could facilitate a constructive critique of published scholarship (12), which could help to ensure readers are informed about the limitations and strengths of a published study (17) or views shared in commentaries about published research (16).

Not surprising, the results of our present study showed ways the write-up of the case article fell short of reporting standards. It is important to note that a variety of factors at play could contribute to the discrepancies with journal article reporting standards, including page restrictions and varying policies between journals in how sections ought to be formatted (34). According to our findings, the abstract section seemed to most visibly illustrate this convergent issue. We observed it was one of two article sections that showed a large discrepancy in the number of standards met between the two table lists used in the present study (i.e., Table 1 versus Table 9: 62.5%-met versus 0%-met, respectively). Interestingly, the conclusion section of the abstract required revision according to both standards. Specifically, the exact implications of the case article’s findings (i.e., a systematic review of the literature), and which were conceptualized within the article’s Discussion section, were not presented (required for Table 1 standards) within the case article’s Abstract. General strengths and limitations identified through the critical appraisal tools used by the authors were not mentioned within the Abstract either (as is required for Table 9 standards).

Abstracts may be the most read section of any research article (4), and they might also be the only section that is (closely) read (4, 27). Thus, omissions there are especially problematic. Readers can misconstrue the results or be left with vague ideas on how a study’s findings may apply to their own line of work (unfortunately, too, abstracts may promote disinformation when they are knowingly presenting, as factual, biased or nefarious research results; for further discussion, see reference number 13). On this latter topic, at least two issues may impair the possible replication or reproducibility of the case article’s findings. First, no a priori analytic plan was reasonably specified within the main text Methods section (34). Second, the aggregated data presented as results were comprised in manner akin to vote counting (8), where practical significance is assumed from the number of significant results and not based on corrected measures of effect (7).

The Introduction section of the case article mirrored the Abstract section. Both sections had the largest discrepancy between the two sets of reporting standards, in terms of the percentage of standards met. In general, more of Table 1 standards were met than Table 9, but a lot more was met for Table 1 regarding the case article’s Abstract and Introduction (e.g., the discrepancy was 9.6% comparing the Results section between the tables, versus a discrepancy of approximately 64.6% for comparing the Abstract/Introduction section). It might baffle readers to see such large gaps between reporting standards within the same applicable article. However, it is important to keep in mind each type of standard has a different educational (think, scholarly) purpose (9). Study designs contribute different types of knowledge and theoretical understanding of complex issues. The association of health literacy with physical activity behavior, the focus of the case article, is a complex issue. This is not only because health literacy is a meta-concept dealing with cognitive and affective domains of literacy (18, 22), but it also imparts hypothesized ways in which health behaviors are affected by a person’s interactions with healthcare systems and related materials (20, 36). The Table 9 standards for the introduction tasks authors to focus on summarizing the issues which affect a general methodological or theoretical understanding of how variables relate. Specific to the case article, this would entail how components of health literacy might associate with physical activity behavior. Thus, although the case article presents a rationale for better understanding why health literacy generally associates with physical activity (e.g., in terms of economic and health burden), the reader does not finish the Introduction having learned about conclusions yet supported by the research literature. Looking to the Discussion section standards for Table 9, the reader finishes the article with little precepts for thinking critically about health literacy’s relationship with physical activity in the context of their own occupation (e.g., as a policymaker, researcher, practitioner). Promoting critical understanding of research trends should be a major objective of any systematic literature review (26).

As evidenced by our findings, the write-up of the case article had several strengths. Looking to the Methods sections, the authors provided a succinct description of their process in obtaining their sample and ensuring good internal validity of their research design. The authors were methodical in their systematic location and filtration of potentially eligible studies for inclusion in their review. They also report inter-rater reliability to the results of their critical appraisal.

Looking to the Results section, the authors succinctly organize their findings into logical sections. Moreover, they provide a well-annotated table describing the scope, methodology, and key findings of each study included in their review. The authors provide an exemplary PRSIMA-style flowchart, showing each stage of their sampling process (6). We also alluded to the fact that if the case article was an observational study, then the write-up of its Introduction section would likely have been fully satisfactory according to Table 1 guidelines.

We can look to the Discussion section, too, to observe additional strengths of the case article’s write-up. Although the authors do not incorporate a comparative discussion of their findings with relevant research, nor substantially incorporate the findings of their systematic critical appraisal in framing their conclusions, they do discuss implications of the descriptive findings of their systematic literature review. They inform the reader on what could be concluded from their finding of similar trends between subjective and objective measures of health literacy, given previous research reports mixed results (10). They also caution the reader to keep in mind that only one study in their sample used a randomized control trial study design. At the very least, the case article provides a general descriptive understanding of how health literacy status may relate to weekly physical activity behavior, as well as suggestions for future research.

Taken together, the findings of this research critique showcase the immense challenge authors face in complying with reporting standards (24, 25). This immense challenge is not limited to the case article of the present study or to kinesiology in general, rather many scientific disciplines seem to be affected (24, 25). One specific implication is that awareness of APA journal article reporting standards is modest, despite many kinesiology and other related journals requiring that authors adhere to APA publication guidelines. Our results demonstrate how performing a post-publication peer review using journal article reporting standards could add to discourse on how to improve the design and reporting of kinesiology research. Professional societies could promote awareness of APA journal article reporting standards in at least two ways: (a) explicitly stating their use is required within instructional material provided to prospective authors and (b) inviting conference presentations based on research critiques using APA reporting standards. In our experience thus far, at least one nationally recognized regional conference does the latter. The Western Society for Kinesiology and Wellness invites student-led research critiques at their annual conference (38), which was the catalyst to the present study (37).

Limitations: There are limitations to this brief report, which should guide the interpretation of its results and future research. First, rater reliability before discussion for consensus ranged between moderate to poor, suggesting it was difficult to determine when elements of the case article had fully met a criterion. While the reported results are based upon a full consensus between the coders (1^st and 2^nd authors) following discussion, and independent verification by an external expert in the field, it is possible researchers replicating this study may have moderately different results. The primary coders were an undergraduate student-faculty dyad, and future research is needed to determine if a similar pre-discussion rater agreement is observed. With that in mind, a possible solution for increasing inter-rater reliability before discussion for consensus is to practice with similar articles first. Pilot-testing the agreed upon coding form, using the same quantified measures of reliability planned for the actual study sample (14), may support acceptable inter-rater reliability before a discussion for consensus is conducted (if needed), even between faculty-student rating teams (33). Finally, this brief report was based on a case-study analysis of one published article. The findings should not be generalized to systematic reviews in the kinesiology and health promotion literature. Future studies that expand the number of articles analyzed or categories used could indicate the degree kinesiology research meets APA journal article reporting standards.

Implications: If authors are taught (and journals require) use of journal article reporting standards, this would ensure that research articles in kinesiology and other fields are complete, comprehensive, and convey the objective(s) of the research study clearly and in a manner consistent with the employed study design (3). For the present study, while the case article’s scope (i.e., a systematic review) was relayed to the end-user, we found key details were missing per APA journal article reporting standards (2, 3). If those details were present within the article, that would help readers finish the article with an informed and critical understanding about the study’s research methodology and findings (e.g., the case article’s Abstract section did not present exact implications for practice or policy based on the findings of the systematic review, the case article’s Abstract section did not summarize the strengths and limitations of studies included in the systematic review based on the authors’ critical appraisal of those studies). Thus, the results of the present study (i.e., of this post-publication peer review) should encourage continuing education on journal article reporting standards for diverse stakeholders, including authors, reviewers, educators, and students. Continuing education on journal article reporting standards, and skill-building in this area, should promote quality research design and reporting practices in kinesiology and elsewhere, allowing for greater transparency in, and comprehension and replication of, the published research literature (11).