How to evaluate a manuscript for publication?

Abstract

This evaluation was undertaken to analyse the overall merit of studies for publication in Medical Journals. Peer review, in which peer experts evaluate the value of a manuscript submitted to a professional journal, is regarded as a crucial step in publication. It helps to ensure that published articles describe experiments that focus on important issues and that the research is well designed and executed. By using previous guidelines and literature review, we have developed an assessment tool to evaluate the scientific studies in an effective and systematic order. Using these tools will facilitate comprehensive assessment and will contribute in generating constructive criticisms.

Introduction

The need for qualified peer reviewers for biomedical journals has never been greater. Between 1966 and 1985, the number of journals and articles indexed by the National Library of Medicine increased by 30% and 73%, respectively (1). Sustained growth of biomedical literature and the staggering increase in drug research and development support the need for health professionals to take an active role in reviewing manuscripts submitted for publication. The technical and scientific merits of manuscripts submitted for publication in a Journal are normally evaluated by an anonymous peer-review process. As with gathering the data, an important concern here is how long the data will take to analyse. The process of analysing and categorising qualitative data can be not only very time intensive but also very subjective as one person may think it is almost perfect and another sees it riddled with problems. However, for the publisher, the evaluation process and its outcome is determined by some simple guidelines such as readability, subject matter (is it the kind of subject the publisher is looking for?), and marketability. Unfortunately, few formal training programs exist for those who are interested in developing their skills in this process (2) and there is limited formal assessment to analyse the peer reviewer performance (3). As with many aspects of clinical medicine, the adage “see one, do one, teach one” often applies to peer review. In this review we evaluate the present literature by generating an effective assessment tool for reviewing process.

What is peer review?

When a manuscript is submitted to a journal for consideration, the journal’s editor is ultimately responsible for its fate. If the manuscript is outside the journal’s interest, the editor may reject it immediately or ask the author to resubmit it with revisions. If the editor considers the manuscript to be appropriate for the journal, the article will undergo internal review by members of the editorial advisory board and/or external peer review. External peer review is considered by most as a crucial part of manuscript evaluation; however, no consensus exists as to its definition. In one definition, peer review is “a negotiation between the author and journal about the scope of knowledge claims that will ultimately appear in print (4). Other interpretations of these definitions are less positive, with peer review being described as subjective, prejudicial, crude, offensive, and secretive (5). Harden’s objective (6) structured clinical examination (OSCE) dramatically changed the assessment of professional competence to evaluate the performance of professional behaviours. There is a need for such a more comprehensive assessment tool to unify all these definitions on peer review (Table 1) (7).

Table 1.

Dissecting the evaluation (Aspinwall 1992)

Goal	Why	To generate an assessment tool for evaluation the value of a manuscript for publication
Aspect	What	Study design
Method	How	Using guidelines and current literature
Sources	Who	Where the study is performed
Timing	When	During which period of time
Results	So what?	Constructive critics

Evaluation Tools

Although readers of the scientific literature must judge the quality of a research article for themselves, the peer-review system is an extremely valuable safeguard. First, it allows readers some degree of confidence regarding the quality of the article, which is particularly important in areas with which they are not familiar. Second, it reduces the time spent reading a paper that fails to conform to generally accepted standards. Thus, it is essential that reviewers carefully evaluate a manuscript, a process that often requires several hours. Each reviewer has a personal method of evaluating a manuscript, and a number of different approaches are described in the literature (8). A thorough evaluation should objectively judge all aspects of the manuscript as mentioned in Table 2. Individuals should take the time necessary to a) thoroughly evaluate a manuscript they have agreed to review b) Consider the quality and significance of the experimental and theoretical work, the completeness of the description of methods and materials, the logical basis of the interpretation of the results, and the exposition with due regard to the maintenance of high standards of communication. c) Evaluations should include constructive suggestions for revision, including, if appropriate, indication of where statements may require additional reference to the published literature.

Table 2.

Check list

Sections	Explanation
Abstract	How participants were allocated to interventions
Objectives and Hypothesis	Specific, Measurable, Achievable, Realistic and Time framed (SMART)
Background	Scientific background and explanation of rationale
Participants	Eligibility criteria for participants
Sample size	How sample size was determined
Recruitment	Dates defining the periods of recruitment and follow-up
Baseline data	Baseline demographic and clinical characteristics of each group
Randomization	Method used to generate the random allocation sequence.
Statistical methods	Statistical methods used to compare groups for primary outcome
Methods	Precise details of the interventions intended for each group and how and when they were actually administered
Results	Flow of participants through each stage
Outcomes	Clearly defined primary and secondary outcome measures

By using these tools and check list in evaluation, I have divided my observation and comments in major and minor. I had one major concern and a few minor with regard to the data interpretation (quality), the structure of this document, their methods and their results respectively. Finally a confidential cover letter to the editor accompanied in which briefly the scientific merit of the manuscript was summarized and appropriate advice was given.

Evaluation process and outcomes

Evaluation of research depends on complete and accurate reporting (9). In this section we briefly describe the assessment criteria. The technique of triangulation was used to generate reliable assessment criteria (10). Evaluation data were selected from various guidelines, and information for reviewers from various Medical Journals we worked with. The weaknesses should be identified by using assessment tools as summarized in Table 2–3. Subsequently, the identified weaknesses might be divided into major as serious and in some cases the major might mean incompatible with further processing the manuscript leading to immediate rejection and minor weaknesses. The major concern is usually related to the quality and significance of the experimental and theoretical work. The manuscript finally might undergo an assessment according to a scoring system as indicated in Table 3.

Table 3.

Evaluation of scientific reliability of research using scoring criteria: 1-5, 1 indicating poor and 5 as excellent for each component of a study. Calculation of final scores will appear as excellent if it score between 20-30 points, as average if between 10-20 and as poor if <10 scores

Research question	Clearly defined and appropriately answered?
Overall design of study	Adequate, relevant
Participants studied	Adequately described and their conditions defined?
Methods	Adequately described? Ethical issues discussed?
Results	Answer the research question? Credible? Well presented?
Interpretation and conclusions	Warranted by and sufficiently derived from/focused on the data? Message clear?
References	Up to date and relevant? Any glaring omissions?

Discussion

Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of these studies are thoroughly and accurately described in published articles (11). Some questions may arise during and after evaluation such as; does this work matter to clinicians, patients, teachers, or policymakers? Having this information, one can determine whether a given manuscript will meet the needs of the readers and be a suitable article for the journal? Therefore, it is also important to review the journal’s mission and know its readership. Information about the journal can be gleaned from “the information for author’s page” or from editorials written by the editor. This is essential in order to determine whether its content is within one’s area of expertise of the journal; to establish whether one might have a conflict of interest; to identify the type of article; and to identify the study hypothesis or its objectives. Does the work add enough to what is already in the published literature? Does the paper read well and make sense (Table 3)?

If the paper is suitable for the journal’s reader with reasonable objectives and outcome, the critical read should be focused on two equally important questions: how relevant or important is this paper, and can the paper be improved, and if so, how? This step usually does not consist of a single read-through. It may require a few reads to answer these questions properly.

In the mid-1990s, two independent initiatives to improve the quality of reports of Randomized Controlled Trials (RCTs) led to the publication of the CONSORT (Consolidated Standards of Reporting Trials) statement (12). This statement was developed by an international group of clinical trialists, statisticians, epidemiologists, and biomedical editors. Preliminary data indicate that the use of CONSORT does indeed help to assess the validity of RCTs results and to improve the quality of reports (13). They are primarily intended for use in writing, reviewing, or evaluating reports of simple two-group, parallel RCTs.

As some other studies far from being transparent, the reporting of this and other related trials have been often incomplete, compounding problems arising from poor methodology (14). Other reasons for the authors’ failure be screening strategy by using less reliable tools (methods) leading to undefined outcomes (Table 2). On the other hand the search for accuracy and reliability must be balanced with the search for innovation. Most agree that a flawed paper can have merit and it is important to not miss this by focusing too narrowly on quality control.

At the end of this evaluation process raise the final question; whether the data provided by the authors are potentially publishable or not. It is not very straight forward and you need to come with a reasonable advice. It may be apparent that some papers will meet the rejection threshold, as they tend to meet this threshold in the methods section. Finally, the promise of qualitative research is more likely to be fulfilled if the health services research community more deeply understands the diversity of qualitative research and the methods criteria for evaluating it. Clear and effective criteria are critical for shaping a qualitative evaluation of a manuscript in as they provide a benchmark against which research can be assessed.

(Please cite as: Rostami K, Khadjooi K, Abasaeed-Elhag R, Ishaq S. How to evaluate a manuscript for publication? Gastroenterology and Hepatology From Bed to Bench 2011;4(2):58-62).