Development of a Critical Appraisal Tool (AIMRDA) for the Peer-Review of Studies Assessing the Anticancer Activity of Natural Products: A Step towards Reproducibility

Abstract

The journal of APJCP (Asian Pacific Journal of Cancer Prevention) focuses to gather relevant and up-to-date novel information’s related to cancer sciences. The research methodologies and approaches adopted by the researcher are prone to variation which may be desirable in the context of novel scientific findings however, the reproducibility for these studies needs to be unified and assured. The reproducibility issues are highly concerned when preclinical studies are reported in cancer, for natural products in particular. The natural products and medicinal plants are prone to a wide variation in terms of phytochemistry and phyto-pharmacology, ultimately affecting the end results for cancer studies. Hence the need for specific guidelines to adopt a best-practice in cancer research are utmost essential. The current AIMRDA guidelines aims to develop a consensus-based tool in order to enhance the quality and assure the reproducibility of studies reporting natural products in cancer prevention. A core working committee of the experts developed an initial draft for the guidelines where more focus was kept for the inclusion of specific items not covered in previous published tools. The initial draft was peer-reviewed, experts-views provided, and improved by a scientific committee comprising of field research experts, editorial experts of different journals, and academics working in different organization worldwide. The feedback from continuous online meetings, mail communications, and webinars resulted a final draft in the shape of a checklist tool, covering the best practices related to the field of natural products research in cancer prevention and treatment. It is mandatory for the authors to read and follow the AIMRDA tool, and be aware of the good-practices to be followed in cancer research prior to any submission to APJCP. Though the tool is developed based on experts in the field, it needs to be further updated and validated in practice via implementation in the field.

Introduction

Reproducibility is the major area of concern in medical research to benefit humans and save financial and opportunity costs. It has been noticed that the retraction of publications from 1975 to 2012 increased ten folds. The cost to the retracted papers between 1992–2012 incurred approximately USD58 million (Stern et al., 2014). The issues of irreproducibility with possible solutions have been highlighted in pre-clinical cancer research in 2011 and 2012 (Van Dang, 2020), which revealed a remarkably higher failure rate for cancer clinical trials when the pre-clinical research protocols were exchanged with clinical trials (Hutchinson et al., 2011). The causes of irreproducibility include; flawed study design, nontransparent reporting of the methods used, incorrect choice of statistical methods, variability in research materials, including mistaken cell line identities, and scientific misconduct (Van Dang, 2020). One of the major sources of irreproducibility is the lack of repeatability of findings, where the identification and responding to flawed research become crucial. To overcome this issue, the development and implementation of enhanced publishing guidelines for journals might be an essential and useful solution, as irreproducibility negatively affects the readership, prestige, reputation as well as increases the administrative costs for retraction and errata (Freedman et al., 2017).

It has been reported that journals with checklist requirements showed improved transparency (Han et al., 2017). The majority of the novel findings published in reputed journals have been found to be non-reproducible, merely due to lack of comprehensive information about the experimental design, reagents quality, methods specifications, etc. The key sources of irreproducibility in these scenarios are improper usage of experimental models, poorly written methodologies (without mentioning variables), and confining to traditional ways of data presentation (Yosten et al., 2018). Hence, the quality of pre-clinical data in published articles may be considered a significant source of failure for cancer clinical trials. The quality of this research work may be optimized by careful consideration of design, analysis, and presentation. “The scientific process demands the highest standards of quality, ethics and rigour” (Begley et al., 2012). It is worth noting that, in response to the importance of reproducibility, various reputed journals have developed and implemented consensus guidelines for reporting pre-clinical studies and their associated data (Principles, 2017).

Every journal has its specific guidelines for reporting and presenting data that may result in some short-term muddle among authors; however, following these guidelines enhances the quality of work presented to the scientific community. The enactment of these strategies could provide long-term benefits in detecting effective methods and extreme accomplishes (Freedman et al., 2017). The journals with appropriate checklist requirements prior to any submission improved the procedural information in pre-clinical studies (Han et al., 2017). Several articles have already discussed and reported the sources of irreproducibility in research findings (Ioannidis, 2005). In addition, the pharmaceutical firms also expressed their concern with the claimed 25% reproducibility of pre-clinical trials (Prinz et al., 2011). The guidelines in the shape of a uniform consensus-based tool development (AIMRDA) for reporting such studies is an appealing way to enhance the quality and present reproducible findings. Herein, a tool has been developed (Figure 1, Table 1) for the APJCP with the objectives:

Figure 1.

Abstract, Introduction, Materials and Method, Results, Discussion and Acknowledgement (AIMRDA)

1) to develop and validate reliable and consensus-based guidelines for authors reporting on research studies evaluating the anticancer activities of natural compounds.

2) to provide an objectives-directed, specific tool for reviewers and editors in order to standardize the evaluation of any manuscript reporting natural products anticancer activities.

Materials and Methods

The road map to develop content and construct validity of this tool (as a quality assessment instrument for reviewers and a quality improvement guideline for authors) included different steps. A core working committee was constructed consisting of experts from the fields of natural products, cancer, ethnopharmacology, ethnobotany, and cell culture techniques for natural products in cancer. An initial draft of the tool was proposed. This draft was circulated among a scientific committee consisting of editors and scientists with relevant expertise in the field of natural compounds and in-vitro studies. The scientific committee members evaluated, revised, and presented their comments for the improvement of the initial draft. The core working committee provided the first draft, managed the scientific committee’s consensus, and finalized the guidelines. In addition, three independent reviewers/editors were involved in evaluating the reliability of the developed tool (based on expertise). The details of all steps are listed as follows:

1- Step one: a “core working committee” of four experts develop the first draft of the guidelines going through the following steps:

a. a comprehensive review of available relevant quality assessment tools (both generic and specific).

b. extracting important and relevant items from the tools and adapting these items in order to develop the guideline’s specific items.

c. evaluating and developing new specific items, not covered in any previous tools.

d. holding several rounds of Delphi and online meetings to finalize the first draft of the tool, based on the selected items of steps b and c.

2- Step two: following the development of the first draft, a “scientific committee” was constructed, including at least 10 editorial experts from different countries, various journals and international societies. The core working committee in communication with scientific committee members obtained the comments and suggestions for the draft item, using a pre-define electronic form which aims to:

a. assess the current items of the first draft of this tool, based on specific criteria of necessity and applicability.

b. suggest any correction or revision to current items.

c. suggest any new items to be added to the first draft

3- Step Three: finally, the core working committee adapted/added the comments/suggestions from scientific committee members, and a 1^st final draft was produced. This draft was presented and further discussed in detail in an online meeting with the respective members of the scientific and core working committee. The semi-final version of the draft was improved through the insertion or deletion of important items in the tool. The consensus developed in the meeting led to the production of a final draft in the form of AIMRDA tool. This tool may be used in further surveys to collect the data for validation of this instrument.

TOOL: ABSTRACT, INTRODUCTION, MATERIALS AND METHOD, RESULTS, DISCUSSION AND ACKNOWLEDGEMENT (AIMRDA)

A1. Title

Provide a clear idea about a suitable and concise title (10 to 20 words) of an article. The title of an article plays an important role in citation, visibility and impression on readers. A crowded title or the use of an abbreviation makes the title less attractive for readers. Furthermore, an appealing title may present the main variables including the name of the natural products (generic or scientific), the histopathologic types of cancer, in vitro model systems, and assessed outcomes.

A2-1 to A2-4: Abstract

Following the administrative assessment phase of the structured abstract, the authors/reviewers will check their articles abstract for clear objectives, methodology, results, and conclusive statements. The first section of a good abstract may provide the gap(s) in research based on which the study was designed. The second section of the abstract needs to explain the main objectives of the work, indicating its novelty and/or difference compared to previous such studies. The third section of the abstract should briefly describe the natural products preparation, thereby indicating the appropriate tools and methodologies used for its extraction, identification/quantification, in vitro model systems, and anticancer assays. The results section discusses the specific main outcomes of the study whereas, the last part of the abstract (conclusion) usually provides a qualitative assessment of the anticancer effect of the natural compound and highlights the importance of the work with future directives, if required.

I1-I3: Introduction

The introduction presents the background of the work in such a way that the readers may find the research work relevant, interesting, well-organized, and valuable. The use of unnecessary or general statements needs to be avoided. The rationale and objectives of the study shall be described in an organized way (Wallwork, 2016). The tool for authors/reviewers divides the introduction into three parts; first part deals with the background of the natural products with their reported phytochemical profile and ethnopharmacological relevance, second part justifies the rationale for the selection of the tested agent as a probable candidate for cancer prevention or treatment (more importantly based on available literature and evidence) whereas, third and last part of the introduction describes the specific objectives of the research to be achieved, mentioning the novelty or confirmatory nature of the work.

M: Materials and Methods

The details of materials and methods need to be in-depth, providing the manufacturer or supplier information, part numbers or models used, dimensions of tools or instruments used (such as chromatography columns), the quality and purity of chemicals or standards used (analytical or standard grade), and proper sequence for the methods applied. All the information needs to be provided to the extent that other scientists could easily repeat/reproduce the same work. The common checklist items (such as cell lines, reagents, animals, human subjects, study, and laboratory protocols) relevant to cancer research can complement a slice to the transparency and reproducibility challenges faced by the community (Van Dang, 2020). There are no length or words limits for the material and methods section to follow. However, step-by-step explanations of protocols or links to protocols are critical for improving the methodology in terms of replication whenever needed (Principles, 2017; Schultz et al, 2020).

M1-M5: Natural product characteristics

Natural products under investigation need to strictly follow the guidelines of AIMRDA tool which consists of; mentioning the proper information regarding identification/taxonomy through experts in the field, time/season and point of collection, processing and storage methods, amount of material collected (g/Kg), extraction method and medium, and quantification of the major active ingredient/s (at least the single main active ingredient with proposed pharmacological/biological property). This may help advance the quality of study as well as its reproducibility. Furthermore, these well-characterized natural products should be subjected to in vitro, in vivo studies or both.

M6-M8: Materials, reagents and software

The quality and identity of research reagents can significantly contribute to reproducibility. The specifications, purity, vendor, and country of origin of materials, reagents, or software used may be provided in order to facilitate the current work’s reproducibility. The chemical impurities in reagents or approved drugs may have a considerable biological impact as MacLeod and colleagues (MacLeod et al., 1999) reported cell line cross-contamination in 252 human tumor cell line cultures, with 18% of the cell lines contaminated with Hela cells. This led to the establishment of the International Cell Line Authentication Committee (https://iclac.org) registry of the contaminated sources in 2012 and the Cellosaurus database (https://web.expasy.org/cellosaurus/) in 2018 covering over 10 x 10⁴ cell lines, and the use of Research Resource Identifiers (https://www.rrids.org) that includes 1.5 million registered antibodies (Van Dang, 2020).

M9-M13

cancer cell lines are invaluable rapid tools for evaluation in basic research due to continuous culturing, countless experiments, and most importantly, fewer regulatory restrictions compared to in vivo models. However, the long-standing reviving and using non-authenticated cell lines may divert the directions and outcomes of research (Jacob et al., 2014). For in vitro model systems, the accurate labelling of cell lines carries prime importance in terms of reproducibility. In most of the reported cell lines studies, wrong labelling with a lack of proper description/cross-contamination were the major sources of irreproducibility, which led to a loss of millions of dollars in research funding (Lorsch et al., 2014). Genotyping, properly labelled, and well-characterized cell lines may solve the mentioned issues associated with cell lines. The development of alternate models for investigation may be another option to be considered for reproducible findings (Bahar et al., 2001; Begley et al., 2012; Eisner, 2018; Im et al., 2011; Jacob et al., 2014). The problem of cross-contamination may be prevented if the trend of simultaneous multiple cell lines culture is avoided and standard cell culturing protocols are followed. The standard laboratory reagents, validated cell cultures, and confirmed cell lines from repositories like ATCC (American type culture collection) may further enhance the replication of results repeated by other scientists (Almeida et al., 2016; Freedman et al., 2015a; Freedman et al., 2015b). For more in-depth information about upgrading the standards and reproducibility of in vitro studies, the authors are encouraged to read the information provided by Hirsch and Schildknecht (2019); Hirsch et al., (2019).

M14

Experimental outcomes: this section clearly defines the primary and secondary experimental outcomes (e.g., survival fraction, growth inhibition, cell migration, angiogenesis, etc.)

M15-M25

Design of experiment: it is necessary to specify the number of biological replications (n) per each intervention and explain; how the number of replications was decided? Additionally, provide the details of sample size calculation; indicate the use of multiple biological entities (more than one cell line, organoid, etc.) from biologically independent sources as experimental units. Otherwise, the authors need to justify the use of a single entity in their experiment. Likewise, the authors need to mention; the procedure for random assignment of experimental units to various groups, the method of randomization, allocation concealment, blinded conduct of the experiment with blinded assessments of outcomes, methods of assessments outcomes reported, concentrations of the test product, and exposure or treatment times applied. In case of variables such as IC₅₀ (GI50) or EC50, it is mandatory to indicate the use of the four-parametric logistic model and the use of at least five concentrations of the test products used to calculate these variables. The use of a vehicle (negative control), an appropriate positive control, and the use of normal biological entities (normal cell lines, normal organoids, etc.) beside neoplastic models, if selective cytotoxicity has been assessed, should be indicated in detail. Express the use of the appropriate method of drug interaction analysis if synergism/antagonism has been assessed.

M26-M29: Statistical analysis

It is important to mention the software or statistical tools used with details of the statistical methods used for each analysis. Similarly, the authors need to report the specific unit of analysis for each dataset, proper identification of the nature of variables used, methods/names of the tests used to assess the dataset with proper F- (F-distribution), P- (significance value), and X² (Chi square) values, etc. Authors should provide appropriate assumptions for the statistical approach used in terms of test/null hypothesis rejection/acceptance. The adoption of wrong experimental design and statistical analysis has been revealed as a significant source of data irreproducibility and negative outcomes (Landis et al, 2012; Sciences, 2015). The authors may agree and provide a statement for datasheets availability or datasets used for statistical tools and their reported outcomes

M30: Ethics code

Every study needs to report and include the protocol approval by the ethics committee of the respective organization, hospital, or University where the research has been conducted. A proper ethical approval number with the approval period needs to be mentioned in the study.

R1-R7: Results

The study results should be presented in a way that should provide a general panoramic view of the experiments without the need to study the methodology in detail. The readers may be invited to look into the logically organized infographic presentations in the form of proper tables, figures, or graphical abstracts, and highlights presented in the article (Wallwork, 2016). Following the AIMRDA tool, the results for the phytochemical profiling of the natural products tested and pictorial presentation are mandatory. For each experimental group, report the most relevant characteristics of the; in vitro model before treatment, the effect of vehicle on in vitro model system, the number of experimental units in each group included in each analysis, and absolute numbers (e.g., 2/4, not 50%), etc. The data unification is of prime importance (both in the text and Tables). For instance, 24.333 and 35.1244 may be presented more appropriately and unified as 24.33 and 35.12 i.e. numbers of the decimals remain the same. A proper explanation may be presented if any data is not included in any analysis. The attrition information for each group and results for each analysis with a measure of precision (e.g., standard error or confidence interval) need to be referred to in the text, and should be expressed in a legible, easy to read, and comprehensible manner. When using high throughput screening for natural products study, all the natural products candidates including the failed one should be listed.

D1-D7: Discussion

The discussion summarizes key points from the results with a parallel reference to the study objectives. Interpret the results, considering the study objectives and hypothesis, current theory, and other relevant studies in the literature. For antiproliferative natural products, interpret that the test agent; has selective cytotoxicity against neoplastic cells, is not anti-life or life-threatening with hazardous effects, and concentrations showing the favorable outcomes in in vitro are suitable for further pharmaceutical development. Furthermore, discuss the mechanisms of action of natural products used, explain the limitations of the study with respect to methodology or findings, if any, and provide suitable comments on whether and how this study’s findings are likely to translate to other biological systems, including any relevance to human cancers. Finally, the discussion may provide a comparative novelty of the current study in parallel to the previous similar studies. It is worthy to provide any future directives based on the outcomes of the current study.

AK1-AK2: Acknowledgement

Enlist all the funding sources (including grant number) and the funder(s) role in the study. Report if the experimental protocol has been registered in the journals or online resources.

Author Contribution Statement

None.