PUBLISHING YOUR WORK IN A JOURNAL: UNDERSTANDING THE PEER REVIEW PROCESS

Abstract

Manuscripts have been subjected to the peer review process prior to publication for over 300 years. Currently, the peer review process is used by almost all scientific journals, and The International Journal of Sports Physical Therapy is no exception. Scholarly publication is the means by which new work is communicated and peer review is an important part of this process. Peer review is a vital part of the quality control mechanism that is used to determine what is published, and what is not. The purpose of this commentary is to provide a description of the peer review process, both generally, and as utilized by The International Journal of Sports Physical Therapy. It is the hope of the authors that this will assist those who submit scholarly works to understand the purpose of the peer review process, as well as to appreciate the length of time required for a manuscript to complete the process and move toward publication.

INTRODUCTION

Manuscripts have been subjected to the peer review process prior to publication for over 300 years. The Royal Societies of Edinburgh and London first began seeking help from their membership with the selection process of articles for their publication in the early to mid‐18^th century.¹ Over time, other professional societies adopted the practice of peer review, however, as the process was introduced it was often disorganized and in most cases depended upon the chief editor. In the middle of the 20^th century, the peer review process became more widespread and standardized.² The main reason for the increased use of the peer review process is rooted in two main factors. The first of these is the proliferation of manuscripts. In the past, editors of new (and existing) journals often had to struggle to collect enough manuscripts to fill the pages of their journals and as such did not need to be selective. Subsequently, as the need for evidence‐based practice has evolved, submissions to scientific journals have increased to the point where editors need to be much more selective in what gets published in their journals. The second reason for the increased use of the peer review process is the explosion of new information and technology. Areas of expertise have expanded to become more specialized and sophisticated. Because of this, editors were no longer able to be experts in all areas and had to seek opinions and advice from others.^1,2 Currently, the peer review process is used by almost all scientific journals. The International Committee of Medical Journal Editors (ICMJE) defines peer review as: “[Peer review is] the critical assessment of manuscripts submitted to journals by experts who are not part of the editorial staff”.³ The purpose of this clinical commentary is to provide a description of the peer review process, both generally, and as utilized by The International Journal of Sports Physical Therapy (IJSPT). It is the hope of the authors that this will assist those who submit scholarly works to understand the purpose of the peer review process, as well as to appreciate the length of time required for a manuscript to complete the process and move toward publication.

WHAT SHOULD PEER REVIEW DO?

Scholarly publication is the means by which new work is communicated and peer review is an important part of this process. Peer review is an important part of the quality control mechanism that is used to determine what is published, and what is not. In the medical community, most scholarly work or research will not be seriously considered until it has been validated by peer review. Furthermore, the peer review process acts as a filter for interest and relevance to the field being targeted by a journal. Therefore, peer review should serve several purposes:⁴

• 1.
  To help select quality articles for publication (filter out studies that have been poorly conceived, designed, and executed) with the selection being based upon:

  • The scientific merit and validity of the article and its methodology

    • Has the research that is being reported been carried out well with no flaws in the design or methodology?
    • Ensure that the work is reported correctly, with acknowledgement of the existing body of work.
    • Ensure that the results presented have been interpreted correctly and all possible interpretations considered.
    • Ensure that the results are not too preliminary or speculative, but at the same time not block the sharing of innovative new research and theories.
  • The relevance of the article to the specific clinical practice – select work that will be the greatest interest to the readership
  • The interest of the topic to the clinical reader
  • The presentation and understandability of the article itself
• 2.
  To improve the manuscript whenever possible.

  • Generally improve the quality and readability of a publication.
• 3.To check against malfeasance within the scientific and clinical community.
• 4.Provide editors with evidence to make judgments as to whether articles meet the selection criteria for their particular publication.

The main functions of the peer review process are to help maintain standards and ensure that the reporting of research work is as truthful and accurate as possible. Peer review contributes to the ongoing process used by individual clinicians to assess what information to believe and what to view with skepticism. This occurs because individual clinicians with varied levels of experience know that a peer reviewed, published manuscript has been reviewed and deemed worthy by others, often with greater or more varied experience than they possess. While most clinicians have the ability to critically read a research manuscript, they cannot be expected to be experts in all areas and make judgments about topics about which they know little.⁵

THE PEER REVIEW PROCESS

The peer review process is similar for all journals, with some variation expected between journals. The procedure described here is the process used by IJSPT with manuscript submissions. Once an author submits a manuscript through the online submission process, it is automatically logged in and checked to make sure that the submission is complete and has been prepared according to the IJSPT submission instructions. At this time a receipt of manuscript acknowledgement is sent to the author to let them know that their manuscript has been received. Each manuscript is then read by an editor (either individually or in consultation) to assess its suitability for the journal according to the guidelines determined by the editorial policy. This is an important step to ensure that (1) the content falls within the scope of the journal, (2) the manuscript follows editorial policy and procedural guidelines, and (3) that it does not contain an unacceptable level of overlap with manuscripts that are already in press. A manuscript could be rejected without additional review for one or more of the previous reasons, and the author notified.

While manuscripts can be rejected without involving additional reviewers, they cannot be accepted for publication without additional review. So if a manuscript is not rejected when first received, it is then sent out for review to a minimum of two additional reviewers who are part of the journal's cadre of reviewers. Review by Associate Editors or staff may compliment this process. Within the medical and scientific communities, debate continues as to the precise form that a peer review should take. The closed review process is the traditional form of peer review adopted by most journals. One prominent area of contention is the subject of blinding. The most common model seems to be the single‐blinded review, in which the reviewer's identities are withheld from the authors but the reviewers are aware who wrote the paper they are evaluating.⁶ This system has been heavily criticized for having the potential for bias because work originating from certain authors, institutions, or geographic regions may be treated more or less critically. The second type of blinding is the double‐blind review. With a double‐blind review the identity of the authors is also masked during the review process. Both the authors and the reviewers are unaware of each other's identity. This type of review has been popularly endorsed in author surveys and is the model employed by the IJSPT.⁶ While the double‐blind process does appear to be a much fairer method of assessment as compared to the single blind review, this peer review process does have some limitations. Manuscripts that draw heavily on the submitting authors previous research may be difficult to mask effectively while still giving the reviewers the information they need to evaluate the study thoroughly.^6,7,8 Since the reviewers are often content experts within a given topic area, they may get enough clues from the citations in the manuscript and/or from their knowledge of the work going on in that topic area to hypothesize as to whom the author may be. Therefore, although it has been suggested that blinding reviewers to author identity leads to better opinions and reviews, this assertion has not been proven in trials.^9,10 Much can be done to help with this problem through careful attention to the manner in which earlier work is referenced in a paper, although some authors may intentionally make their identity easier to discern if they feel their reputation (and citing their previous publications liberally) will garner better treatment from the reviewers.

Once reviewers are chosen and they accept their review assignment, the real process begins. Most reviewers use some form of checklist that covers some or all of the considerations offered in Appendix 1. Note that this checklist is best utilized with papers that are submitted in the category of Original Research, and different criteria or salient points for assessment may be utilized for other types of submissions such as Case Reports, Clinical Commentaries, and Clinical Suggestions.

The reviewers return their recommendations and reports to the editor (via the online submission system), who assesses them collectively, and then makes a decision, either on his or her own or in consultation with other editors on whether to reject the manuscript (either outright or with encouragement to resubmit), to withhold judgment pending major or minor revisions, to accept it pending satisfactorily completed revisions, or to accept it as written. Rarely, if ever, is a manuscript accepted as written! For manuscripts accepted pending revision, the authors must submit a revised manuscript that will go through all or some of the stages above. Once a manuscript has been revised satisfactorily (more than one revision may or may not be allowed) it will be accepted and put into the production process to be prepared for publication. An outline of this process can be seen in Figure 1. Despite the apparent simplicity in this process, the actual steps may be quite elaborate and involve a number of people and alternative procedures, thus requiring substantial time to complete.

Figure 1.

A graphic display of the “path” a manuscript takes after submission to The International Journal of Sports Physical Therapy.

CONCLUSION

While the peer review process is unlikely to change the basic nature of a given submission, in many cases the authors may add analysis or results, clarify thoughts or parameters, revise the statistical testing methods, increase the number of subjects, or lengthen the time of clinical follow‐up in response to reviewer's requests. Most typically, thoughtful comments provided by reviewers lead to improvements in the presentation of the work in several ways: clarity in writing and descriptions are enhanced, relevant literature is discussed more thoroughly, limitations of methodology are acknowledged, and broad or over‐reaching conclusions are moderated. This can only happen when knowledgeable reviewers take time to participate in the peer review process and evaluate submissions with care and sensitivity. The editors and reviewers of IJSPT are committed to utilization of a stringent yet fair review process in order to assist those who submit scholarly work for publication.

APPENDIX 1: SAMPLE REVIEW GUIDELINES

Title: Does it accurately reflects the purpose, design, results, and conclusions of the study?

Abstract: Does it correctly and succinctly summarize the salient points of the study?

Introduction: Does it provide adequate background and rationale for performing the study?

• Does it place the study in the perspective of research conducted previously in the field?

  • Why is study being done? Identify controversy?
• Is the literature discussed in the introduction adequate to introduce the purpose of the manuscript?

  • Is the functional, biological, and/or clinical significant of the topic established.
  • Strengths and limitations described such that a need for further study is established.

• Is the literature discussed in the introduction directly related to the purpose of the manuscript and necessary to introduce the topic?

• Is it clear how the experimental approach to be used in the present study is likely to yield more definitive or unique insight than previous studies?

• Does it clearly state or imply the study hypothesis(es) or null hypothesis?

• Are the outcomes to be measured clearly described in the introduction or methods section?

• Does the introduction adequately introduce the purpose of the manuscript in a logically compelling way?

• Is a clear and strong rationale provided for the importance of this manuscript?

Study design and methodology: Is the sample described in appropriate detail; procedures and data analysis described clearly and in sufficient detail?

• IRB approved?

• Type of study described? (RCT, Cohort, Case controlled, Case report, etc)

• Is the experimental design of the study capable of answering the question implied by the study hypothesis?

  • Do the methods address the purpose?
• Is there a control or comparison group in the treatment study?

  • Are there factors not controlled between the groups: (list)

• Is the study: Prospective or Retrospective

• Is the methodology described in sufficient detail for others to repeat study

  • Is it reproducible?
  • If not, do the authors provide a proper (peer reviewed) reference that would provide such details?

• Is there a rationale for the experimental design?

• Is the Study Population clearly identified

  • Identified and appropriate to answer question?
  • Informed consent obtained?
  • Admission criteria clearly specified?

    • Inclusion / exclusion criteria
  • Power analysis provided?

    • Where enough subjects studied to detect a difference?
  • Were subjects randomized?

    • What methods were used?
    • If subjects were not randomized, were subjects and controls equivalent?
    • Was the randomization assignment concealed from both patients and healthcare staff until recruitment was complete and irrevocable?
  • Will the subject population allow extensive or rather limited generalizability?
• External validity:

  • Were the subjects asked to participate in the study representative of the entire population from which they were recruited?
  • Were those subjects who were prepared to participate representative of the entire population from which they were recruited?
  • Were the staff, places, and facilities where the patients were treated representative of the treatment the majority of patients received
• Internal validity

  • Was an attempt to blind study subjects to the intervention they have received?
  • Was there an attempt made to blind those measuring the main outcomes of the intervention?

    • Blinding

      • Single‐blind (patient)
      • Double‐blind (patient & investigator)
  • If any of the results of the study were biased on the data dredging, was this made clear?

    • Any analysis that had not been planned at the outset of the study should be clearly indicated.

• Therapeutic intervention clearly defined? Treatments should be clearly described.

• Measurement Instrument or method clearly described?

  • Standard accepted measurement instrument or method? (ie. Universal?)
  • Are metrics provided for standard instruments, procedures, or methods?
  • Non‐standard

    • Unbiased?
    • Validated?
    • Reproducible?

• Are the details as to how the data were derived (calculated) adequately explained so that they can be confirmed by the reviewer and reproduced by future investigators?

• Is it clear how the data will be interpreted to either support or refute the hypothesis?

• Have the characteristics of patients lost to follow‐up been described. Follow‐up

  • Adequate length?

    • Minimal_____ Average_____
  • Is mechanism of follow‐up described?
  • Loss to follow‐up reported?

Soundness of the Results: the outcome of the statistical analysis are presented appropriately and interpreted accurately.

• Are the data reported in a clear, concise, and well‐organized manner?

  • Is there excessive variability in one or more of the measurements for a particular condition compared with others?

• Are the main findings of the study clearly described? Simple outcome data should be reported for all major findings so that the reader can check the major analyses and conclusions.

• All results must be proposed in the methods.

  • Are they relevant to the study or research problem?
  • Are data presented that was not described in the methods?
• Reported in sufficient detail?

  • Statistical results tell statistical significance?
  • Actual results tell clinical significance?

• Was compliance with the intervention reliable?

• Do the tables and figures clarify or confuse?

  • Are all the figures and tables needed?
  • Are the tables and figures properly labeled with titles and the correct units?
  • Is the scaling of the figures appropriate and unbiased?

• Was randomization successful?

• Statistics:

  • Appropriate test(s) chosen?
  • Appropriate p‐value chosen (a priori)?

    • Have the actual probability values been reported rather than <0.05 for the main outcomes except were the probability value is less than 0.001.
    • Have adjustments been made for multiple comparisons?

• Does the study provide estimates of the random variability in the data for the main outcomes?

• Does the analysis adjust for different lengths of follow‐up of patients, or in case‐controlled studies, is the time period between the intervention and outcome the same for cases and controls?

• If findings are negative, was a sufficiently large population studied?

  • Remember: failure to show a difference is NOT the same as showing that there is no difference – may be a lack of power.

• Have all the important adverse events that may be a consequence of the intervention been reported?

• Are findings clinically significant?

  • How do the group differences or responses shown compare with the measurement variability?

Discussion and Conclusion: The implications of the study are consistent with the purpose, methods, and data analysis.

Discussion

• Are the major new findings of the study clearly described and properly emphasized?

  • Is the significance of the present results described?
  • Is it clear how the findings extend previous knowledge in a meaningful way?

• Does it point out weaknesses/limitations of the study?

• Biases:

  • Selection
  • Performance
  • Detection (measurement)
  • Transfer (loss of follow‐up)

• Does it point out the strengths of the study?

• Does it place the study in perspective with existing literature?

  • Discuss similarities and differences
  • Are important experimental observations from previous reports described in the context of the present results?
• Excessive speculation?

  • Does it distinguish author opinion from the conclusions
  • Do the authors support their statements with appropriate references?
  • Do the authors discuss their data in a manner that provides insight beyond that presented in previous sections?

• Is there any other way to interpret and/or explain the data other than that suggested by the authors?

Conclusion

• Was hypothesis proved?

  • Is it based on the data described in the results?
  • Key conclusions adequately supported by the experimental data?

• Does it point out the clinical significance of the conclusions?

• Does it suggest the possible direction of future investigation?

  • Do authors make suggestions as to how the results of their study need to be extended in the future to learn more about the issue in question?
• Are conclusions justified by the results of the study?

  • Does it stray beyond the boundaries of the study?

Organization and Style

• Is the manuscript concise?

  • Is the material presented, without excessive jargon?
  • Are all the graphs or charts needed?

• Was the paper well written, properly organized, and easy to follow?

• Was proper grammar, spelling, and punctuation used throughout?

• Should manuscript be shortened?

• Should manuscript be more comprehensive?

References

• Are the major references included?

• Are all references cited completely and in the desired format of the journal

• References chosen directly relate to the study?

• Avoids secondhand or abstract reference sources?

• Are all references cited correctly in text, e.g superscripted following punctuation.

Overall Significance and Suitability

• Is the manuscript sophisticated enough for the intended professional audience?

  • Was the information presented in an open‐minded and objective manner?

• Is the experimental question significant?

• Is a clear and testable hypothesis presented?

  • Overall method is valid?

• Results are properly presented and believable?

• Conclusions are reasonable on the basis of the results obtained?

• Does manuscript contain new findings or ideas?

• Does the manuscript provide a unique contribution?

  • If not, does it present old material better?