Skip to main content
NCBI home page
PLOS One logoLink to PLOS One
. 2022 Feb 22;17(2):e0250754. doi: 10.1371/journal.pone.0250754

From viability to cell death: Claims with insufficient evidence in high-impact cell culture studies

Ali Burak Özkaya 1,*, Caner Geyik 2,3
Editor: Hamidreza Montazeri Aliabadi4
PMCID: PMC8863264  PMID: 35192623

Abstract

Background

Reliability of preclinical research is of critical concern. Prior studies have demonstrated the low reproducibility of research results and recommend implementing higher standards to improve overall quality and robustness of research. One understudied aspect of this quality issue is the harmony between the research hypotheses and the experimental design in published work.

Methods and findings

In this study we focused on highly cited cell culture studies and investigated whether commonly asserted cell culture claims such as viability, cytotoxicity, proliferation rate, cell death and apoptosis are backed with sufficient experimental evidence or not. We created an open access database containing 280 claims asserted by 103 different high-impact articles as well as the results of this study. Our findings revealed that only 64% of all claims were sufficiently supported by evidence and there were concerning misinterpretations such as considering the results of tetrazolium salt reduction assays as indicators of cell death or apoptosis.

Conclusions

Our analysis revealed a discordance between experimental findings and the way they were presented and discussed in the manuscripts. To improve quality of pre-clinical research, we require clear nomenclature by which different cell culture claims are distinctively categorized; materials and methods sections to be written more meticulously; and cell culture methods to be selected and utilized more carefully. In this paper we recommend a nomenclature for selected cell culture claims as well as a methodology for collecting evidence to support those claims.

Introduction

An alarming concern exists regarding reliability of the published research findings [1, 2]. This is particularly evident in preclinical studies, as clinical translation is minimal [3]. Such low efficiency in research has been discussed extensively in recent years and the lack of reproducibility and overall quality are agreed upon as the main culprits of the problem [4]. Reproducibility in preclinical research is estimated to be between 10% to 25% [5, 6] and the cost of irreproducible research is calculated to be at least 28 billion USD/year in USA alone [7]. There are many factors contributing to this crisis, including lack of robustness, biased design, use of inadequate models (cell line and/or animal), underpowered studies (insufficient sample size), lack of proper controls (i.e. samples that are expected to produce [positive control] or expected not to produce results [negative control], poor use of statistics and the absence of replication/confirmation studies [8]. It is important to note that these design problems often extend to questionable research practices such as p-hacking (performing multiple different statistical analyses just to obtain significance) and cherry-picking (concealing inconsistent or contradictory findings).

Scientists agree that the standards for publishing preclinical research must be raised in such a way as to encourage robustness and rigor [6, 9]. Therefore, many aspects of the preclinical study design have been tackled by various studies over the years [7, 8]. However, the question of whether we can trust results of published preclinical studies remain at large. Even though understanding key concepts and methods for reporting data has been suggested as critical to preserving scientific findings [10], one important aspect of the process, the compatibility of the way the manuscript was written with the actual experimental design, has been overlooked. More specifically, the relationship between the claims of the studies and the evidence provided to support these claims remains underexamined. In this study we investigated if the evidence provided by high-impact studies sufficiently supports cell culture claims authors asserted in their manuscript. It is important to note that we focused only on selected commonly used cell culture claims and our analysis does not cover all the asserted claims. In many cases cell culture may be a small part of the study with minimal effect to its conclusions. Accordingly, our findings should not be interpreted as a measure of article quality.

While we were trying to select the claims to include in our analysis, one of the first things we have noticed was the inconsistency in the nomenclature. Many claims such as cytotoxicity, viability, growth, and proliferation were used interchangeably by the authors. Moreover, there were several publications in which only one type of evidence (tetrazolium reduction assay results) was provided to support various claims. When we searched the literature to refine the consensus nomenclature, much to our disappointment, we could not find any. Possibly, many of these terms were not considered uncommon, unfamiliar, or vague enough to be defined in high-impact reviews or guidelines, or to be included in the glossary sections of the molecular biology, biochemistry and even cell culture textbooks. Therefore, in the current work, we propose a series of definitions and recommendations mostly based on different sections of “Guidance Document on Good In Vitro Method Practices” by OECD [11] which was the only document we find that might be considered as a consensus nomenclature source. Based on these definitions, we analyzed high-impact cell culture studies to investigate if the asserted claims are justifiably backed with evidence or not.

Methods

The study consisted of three phases. In phase one, we selected high-impact cell culture studies. In phase two, we identified some of the cell culture claims asserted by the authors as well as the evidence provided by them to support these claims. In the final phase, we analyzed the sufficiency of the evidence for each of the claims.

Article selection

We searched Web of Science (WOS) database (Clarivate Analytics) for studies that contain at least one of these keywords: “cytotoxicity, viability, cell death, growth inhibition, proliferation, or anti-cancer”. We included original research articles using in vitro techniques. The search string below was used in advanced search feature of WOS:

WOS core database (TS = ("cytotoxicity" OR "viability" OR "cell death" OR "growth inhibition" OR "growth inhibitory" OR "proliferation" OR "anti cancer") AND TS = ("cell culture" OR "in vitro" OR "cell line")) AND LANGUAGE:(English) AND DOCUMENT TYPES: (Article)

Studies published in 2017 and 2018 were retrieved in 10.03.2020. We exported the data as an Excel file, sorted the list based on citations received and selected the most cited 121 publications (each receiving at least 65 citations) as high-impact studies for further analysis. Upon investigation, we excluded 18 articles, including those not containing a claim, those not carried out in cell culture, and those that were review articles and perspectives. After the exclusion process, 103 studies remained for claim analysis.

Claim selection and definitions

Using any one of six distinct terms to present or discuss a finding in the studies was considered a claim. We identified these as: proliferation rate, viability, cell death, apoptosis, cytotoxicity, and cell growth. We outlined our definitions for these claims as follows by using OECD guide GIVIMP [11]:

Proliferation rate represents how fast a group of cells divide over time. To provide sufficient evidence for proliferation rate, an end-point measurement must be able to differentiate the change in division capabilities of the cells from cell death. If the treatment of question induces cell death in the treatment group, there would be fewer living cells (compared to untreated control) without a decrease in proliferation rate. Accordingly, methods specifically focusing on replication rate (such as nucleotide incorporation) or measuring the number of viable cells without a treatment over time as well as real-time observations, and proliferation markers were considered as sufficient evidence.

Viability represents the number of living cells. It is the broadest term since there is no specification regarding the factor affecting the number (such as proliferation rate or cell death). Any method directly measuring the number of living cells and methods measuring metabolic activity or total protein content were considered as sufficient evidence for viability.

Since cell death, by definition, requires cells to die, end point analysis measuring the abundance of living cells cannot provide sufficient evidence for this claim as the measurement does not differentiate the decrease in number due to dying cells from slowed-down proliferation rate. In some cases, when the viability of treated cells is so low that the decrease in proliferation rate is not sufficient to explain the viability loss, cell-death can be assumed. However, even in those cases it is impossible to detect individual contributions of cell-death and decreased proliferation rate to reduced viability. Therefore, only assays measuring death-related alterations such as membrane integrity loss or cell-death specific markers were considered to provide sufficient evidence.

Apoptosis is a form of programmed cell death which has well established and characterized distinct features. The methods that can demonstrate changes in features such as exposure of phosphatidylserine on the outer membrane, DNA fragmentation, morphological changes (nuclear condensation and membrane blebbing) and molecular switches (caspase activation via cleavage) resulting in apoptosis were considered as sufficient evidence to show apoptosis.

Cytotoxicity indicates being toxic to cells. Being toxic itself is a broad term and there are conflicting definitions in use. We decided to consider it as cell death instead of decreased viability as the most widely accepted capability of a toxic agent is killing (as in cytotoxic T cells and cytotoxic chemotherapy), and accepted evidence indicating cell death as sufficient to prove cytotoxicity. This decision had an impact on the final analysis as many of the articles might have used the term to represent viability decrease. We addressed this in results section.

Cell growth may indicate either proliferation rate or cell size change depending on the definition embraced. Since, there already is a term representing proliferation rate as the name implies, we first considered to accept growth as a measure of increased cell volume. However, after investigating the articles in our list, we realized that the term was exclusively used to indicate proliferation rate and consequently we embraced that definition in our analysis.

Database construction

We constructed a database in Airtable to carry out evidence analysis. Information from WOS database including “article name”, “DOI”, “citation count”, “journal name” as well as our parameters of interest including “claim”, “evidence”, “method”, “sufficiency of evidence”, and “subject area” were entered for every article investigated.

Here, “method” represents scientific methods used in the study whereas “evidence” is defined as a supergroup of methods measuring same biological phenomenon. For example, two separate methods such as lactate dehydrogenase (LDH) activity assay and PI both of which measure membrane damage as an indicator of cell death were classified into “membrane integrity” evidence supergroup. Similarly, various tetrazolium and resazurin reduction assays were considered to provide “dehydrogenase activity” evidence as an indicator of cellular metabolic activity.

We have also divided the studies in two notional groups of “subject area” based on field information provided by WOS. The first one is “Biochemistry, Molecular Biology, Genetics, and Medicine” and the second one is “Chemistry, Chemical/Biomedical Engineering, and Materials Science”.

The database is accessible via the link: https://airtable.com/shrClTE87e1l28ExG and its contents are also accessible as S1 Data (Database CSV spread sheet).

Evidence analysis

Evidence was analyzed for each of the claims using a case-by-case approach. We refer to the definitions we have embraced and the OECD guide GIVIMP [11] to determine if the measured parameter provides sufficient evidence for that claim. Table 1 summarizes the claims asserted and the evidence provided by the articles we investigated. The table also provides information regarding our decision on whether a type of evidence is sufficient for a claim or not as well as the rationale behind that decision.

Table 1. Claims and the evidence provided by the articles along with the sufficiency of the evidence and the rationale for our decision.

Claim Evidence Methods/Assays Used Sufficient Rationale
Apoptosis Caspase-3 activity Caspase-3 activity assay, Cleaved caspase-3 protein measurement Yes Molecular switch resulting in apoptosis
Cell morphology Electron microscopy Yes Established feature/characteristic of apoptosis
DNA fragmentation Acridine orange staining, TUNEL assay Yes Established feature/characteristic of apoptosis
Phosphatidylserine exposure Annexin-V staining Yes Established feature/characteristic of apoptosis
Dehydrogenase activity Tetrazolium reduction assay No Indicator of viability but not apoptosis
Membrane integrity Propidium iodide staining No Indicator of cell death but not apoptosis
Bcl-2 Family expression Protein expression measurement No Indicator of a pro-apoptotic signal but not of the apoptosis itself.
Cell Death Membrane integrity Staining by: DAPI, Ethidium bromide, Ethidium homodimer-1, LDH activity, Propidium iodide, Zombie UV Yes Indicates dying/death cells
Phosphatidylserine exposure Annexin-V staining Yes Established feature of apoptosis and accordingly cell death
Cell count Real-time cell imaging No No distinction between cell-death and slowed-down proliferation rate
Dehydrogenase activity Tetrazolium reduction assay No
Esterase activity Calcein AM staining No
Cytotoxicity Membrane integrity Staining by: 7-AAD, Ethidium homodimer-1, Hemolysis assay, LDH activity, Propidium iodide Yes Indicates dying/death cells
Phosphatidylserine exposure Annexin-V staining Yes Established feature of apoptosis and accordingly cell death
Dehydrogenase activity Tetrazolium reduction assay, Resazurin reduction assay No No distinction between cell-death and slowed-down proliferation rate
Cell Growth Cell count Real-time cell imaging Yes Indicates cell division and proliferation
Colony formation Crystal violet assay Yes Indicates cell division and proliferation
Nucleotide incorporation BrdU assay Yes Indicates cell division and proliferation
Dehydrogenase activity Tetrazolium reduction assay No No distinction between cell-death and slowed-down proliferation rate
Claim Evidence Methods/Assays Used Sufficient Rationale
Proliferation Dye inclusion Celltrace violet assay Yes Dye dilution over generations indicates proliferation
Ki67 expression Expression at protein level Yes Direct marker for cell proliferation
Nucleotide incorporation [H3]-Thymidine, BrdU, EdU Yes Direct marker for cell proliferation via replication
Phospho-histone H3 Anti-Ph3 staining Yes Lack of Ph3 is an indicator of non-proliferating cells
Cell count Cell analyzer, Cell and particle counter, Unknown Conditional Since these methods cannot identify the reason for viability change, it is considered sufficient only if there is no treatment that may cause cell death. In all other cases it is considered insufficient.
Colony formation Crystal violet assay Conditional
Dehydrogenase activity Tetrazolium reduction assay, Resazurin reduction assay Conditional
DNA amount Picogreen staining Conditional
Actin Phalloidin staining No Actin visualization is not an indicator of proliferation
Esterase activity Fluorescein DA staining No No distinction between cell-death and slowed-down proliferation rate
Signaling Pathways Protein expression measurement No May indicate activation of growth-related signaling pathways but does not provide evidence for proliferation rate.
Viability ATP amount Luciferase-based assay Yes ATP amount correlates with the number of viable cells
Dehydrogenase activity Tetrazolium reduction assay, Resazurin reduction assay Yes DH activity correlates with the number of viable cells
Dye inclusion Staining by: Acridine orange, Hoechst, GhostDye Yes Indicates cells with intact cell membrane
Esterase activity Calcein AM staining, Fluorescein DA staining Yes Indicates cells with intact cell membrane and active esterase
Total protein Sulforhodamine B assay Yes Indicates viable cells
Cell count Cell Analyzer, MAP2 staining, Unknown Conditional Imaging of living cells were considered sufficient. If the method was not mentioned in the article, it was considered as insufficient evidence.
Membrane integrity Staining by: 7-AAD, Propidium iodide, Trypan blue Conditional Utilization of cell impermeable cytotoxicity dyes considered insufficient. (7-AAD and PI).
Trypan blue was an exception since it is possible to observe viable cells directly.
Signaling Pathways Protein expression measurement No May indicate activation of growth-related signaling pathways but does not provide evidence for viability.
Unknown N/A No N/A
Open in a new tab

If there were multiple types of evidence for a single claim, we focused on the strongest of that evidence. For example, even though expression of Bcl-2 members (such as Bax/Bcl-2 ratio) is commonly investigated along with other markers of apoptosis, it is classified as insufficient because it does not provide proper evidence for apoptosis by itself. However, if an established feature of apoptosis (like DNA fragmentation) was demonstrated along with Bcl-2 expression results, we focused only on that evidence, classified it as sufficient and ignored the rest. Another example to this is citing viability assay results as evidence for cell death induction or proliferation rate change when there is further evidence. Normally by itself, viability assay results do not provide information regarding to the cause of the viability change. However, if the mechanism was elucidated via further experimentation and those experiments were cited while making the claim, we then focused only on those experiments instead of the viability assay.

There were some cases where the same type of evidence was determined to be sufficient or insufficient based on the experimental design. Viability indicators such as cell count, colony formation, DNA amount, and dehydrogenase activity were considered to provide sufficient evidence for proliferation only if there was no treatment (hence no reason for a change in cell death ratio). Moreover, if the method utilized to obtain the evidence was not mentioned in the article (or in the S1 Data), we then classified it as insufficient (independent from the actual evidence) as was the case for cell count providing insufficient evidence for viability and proliferation [12].

Another such deviation was membrane integrity as evidence for viability. Normally we considered these cell-death markers (7-AAD and PI) as insufficient, due to the fact they do not indicate living cells by themselves. However, since it is not only possible but quite common to observe cells under microscope after trypan staining, we decided to consider it as sufficient evidence for viability.

The effects of the journal and subject area on evidence sufficiency

After the sufficiency of evidence was assessed for each claim, we then sought to find whether being published in a specific journal or in a subject area would affect sufficiency rate. Seven journals were selected for analysis as they meet our criteria of having at least 10 claims, namely Biomaterials, Cell, Cell Death Dis., Mol. Cancer, Nature, Oncotarget, Ann. Biomed. Eng. (96 claims out of 280, 34.28%). Every journal was compared to the complete data set excluding themselves. Two subject areas: “Biochemistry, Molecular Biology, Genetics, and Medicine” and “Chemistry, Chemical/Biomedical Engineering, and Materials Science” were compared to each other. Fisher’s exact test (2-tail) were used in all comparisons.

Results

We have investigated 280 claims asserted by 103 different high-impact articles. We identified 6 unique claims supported by 20 types of evidence all of which was obtained via 38 different methods. Details are presented in the database (accessible via: https://airtable.com/shrClTE87e1l28ExG or as a S1 Data).

The most common claim was proliferation rate changes with 84 claims, followed by viability, apoptosis, cytotoxicity, cell death, and growth changes (Fig 1A). Upon investigation, we considered the evidence of 102 claims (36%), which was asserted by 67 different studies (65%) as insufficient (Fig 1B). Claims of cytotoxicity (11 sufficient in 37 total claims, 30%), proliferation (34 in 84, 40%), and cell growth (10 in 18 claims, 56%) particularly lacked proper evidence. Viability (64 in 71, 90%), apoptosis (46 in 51, 90%), and cell death (15 in 19, 79%), on the other hand, were more frequently claimed with proper evidence (Fig 1A).

Fig 1. Evidence sufficiency analysis results.

Fig 1

Open in a new tab

1A. Evidence sufficiency according to the claim type; 1B. The number of claims with sufficient or insufficient evidence; 1C. Evidence sufficiency of claims supported by dehydrogenase activity assay; 1D. Evidence sufficiency according to the article subject area.

In case of cytotoxicity, there is a possibility that the authors may have used the term to indicate decrease in cell viability instead of the definition we embraced (cell death). If we had considered cytotoxicity as a measure of viability instead of cell death, all the cytotoxicity claims would have had sufficient evidence increasing the overall sufficiency rate from 64% to 73%.

Measurement of dehydrogenase activity was by far the most common type of evidence. 120 different claims (43% of all claims) including viability, proliferation, cytotoxicity, cell growth, cell death, and even apoptosis put forward dehydrogenase activity findings as evidence (Fig 1C). The most common method used to measure dehydrogenase activity was tetrazolium reduction assay supporting 114 different claims (41% of all claims). Measurement of dehydrogenase activity was followed by measurement of phosphatidylserine exposure (31 claims) and membrane integrity (28 claims).

While the results of dehydrogenase activity assays were interpreted correctly only in 48 claims (48 in 120, 40%), the second most common evidence, measurement of phosphatidylserine exposure, was interpreted correctly (as an indicator of apoptosis) in all related claims (31 in 31, 100%). Similarly, measurement of membrane integrity was mostly correctly utilized as evidence (25 in 28, 89%) (Fig 1C).

We also analyzed whether the subject area or the journal that the article was published in might be an indicator of evidence claim relationship. According to our analysis, neither the subject area (Fig 1D) nor the journal, have significant influence over evidence sufficiency.

Discussion

There is a valid concern regarding reliability of preclinical research and many agree that we need strategies in place to improve the standards [6, 9]. There is a need for a comprehensive guideline for in vitro studies, which clearly and distinctively defines commonly used cell culture terms such as viability, cytotoxicity, proliferation, and cell death along with the correct methodology to measure them. Here, we offer a nomenclature and methodology recommendation for cell culture studies with such claims.

According to the definitions we embraced, viability represents the number of living cells and when there is a decrease in cell viability, there are two possible causes: decrease in how fast a group of cells divide over time (proliferation rate decrease) or induction of cell-death. Cytotoxicity represents reduction of viability via cell-death. Cytostaticity or inhibition of cell growth represents reduction of viability via decreased proliferation rate. Accordingly, methods measuring the number of living cells are appropriate for determining viability and methods detecting dying/dead cells are required to assert claims of cytotoxicity or cell-death. On the other hand, to claim a proliferation change, one must be able to determine the change in number of viable cells while taking cell-death into account.

In this study we created a database containing the data from highly-cited cell culture studies and analyzed the data based on our definitions. Since we focused only on the list of claims we have selected and defined, our analysis is neither an indicator of article quality nor an evaluation of all the claims of the articles. Our findings revealed a discordance between the cell culture claims and the evidence of these studies. This was especially evident in studies utilizing the findings of tetrazolium reduction assay, an indicator of cellular metabolic activity, alone to support various claims. Striking examples include article id#9 [13] claiming viability, proliferation, cytotoxicity and apoptosis changes, and article id#26 [14] claiming viability, proliferation, and growth changes with results from this assay. This is partly due to these assay kits being advertised by their manufacturers as a tool to measure viability, cytotoxicity, proliferation and growth. Combining this with being relatively easier to perform and affordable leads to these assays being perceived as a one-size-fits-all solution by research groups wishing to avoid more complicated cell culture methods. However, this reductionist approach makes it difficult for the findings obtained from the study (assay results) to provide a meaningful answer to the research question (does this treatment kill cells?) of the article. When this approach is embraced by a high impact work, its negative effects may extend beyond its own reliability.

In fact, the articles we analyzed were cited more than 9000 times as of December 2021 (within two to four years). Admittedly, claims without sufficient evidence may not be the reason for citation in most of the cases as cell culture may have a small part in the study. However, the impact of such influential studies with unreliable findings on preclinical research is undeniably large. Defining distinct cell culture claims and adequate methodology to measure them is the first step to increase reliability of similar studies. However, that first step may not be enough if we ignore other issues surrounding basic cell culture techniques.

Even though, in this study we considered measurement of metabolic activity as an indicator of viability as it is by far the most preferred viability assay [15], measuring metabolic activity as an indicator of cell viability must be challenged. Formazan production in tetrazolium reduction assays has been demonstrated to be influenced by multiple factors [16] and the reaction was suggested to be taking place in the plasma membrane due to activity of trans-plasma membrane electron transport [17] instead of cytoplasm or mitochondria due to activity dehydrogenase enzymes. Moreover, inhibition of dehydrogenase activity may not always indicate a loss of viability as the treatment may alter enzymatic activity without affecting the number of living cells and it is suggested to use proper controls to compensate for the effects on metabolism [15]. There are also concerns regarding the Ki67 expression as an indicator of proliferation rate as the protein levels were demonstrated to be influenced by the time cells spent in G0 [18].

In this study we focused on how common cell culture methods were interpreted and whether they could provide sufficient evidence for related claims. However, we did not investigate how each experiment was designed or conducted. Since the strength of evidence depends on the robustness of the design, even if the method chosen is appropriate, it can provide reliable evidence only if it is carried out properly. General recommendations on good cell culture practices such as cell line identification, contaminant screening, proper use of controls (negative, positive and vehicle), appropriate handling and storage of assay components as well as information about the factors interfering assay results (such as cell confluency, media components and plastic materials) are covered by OECD guide GIVIMP [11]. Other recommendations we would like to mention include: proper data normalization (such as normalization for basal cell death in proliferation rate measurement), investigators being blind for the analyses [19, 20], proper data visualization (such as using logarithmic axis plots to demonstrate cell growth) and accounting interfering factors (such as artifacts in high-throughput assay systems [21]).

In this work we offered a nomenclature recommendation by which the most common claims in cell culture studies may be distinctively expressed. As the findings of our study indicate, we believe a more meticulously written materials and methods section and careful selection and utilization of cell culture methods are critical to increase overall quality of preclinical research.

Supporting information

S1 Data

(CSV)

Click here for additional data file. (77.6KB, csv)

Acknowledgments

We would like to thank Yasemin SEVAL ÇELİK for her contributions in revising this manuscript. We would also like to extend our thanks to the reviewers, comments of whom has improved the quality of this paper substantially.

Data Availability

All relevant data are within the manuscript and its Supporting Information files.

Funding Statement

The author(s) received no specific funding for this work.

References

  • 1.Ioannidis JPA. Why Most Published Research Findings Are False. PLOS Med. 2005;2: e124. doi: 10.1371/journal.pmed.0020124 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016;533: 452–454. doi: 10.1038/533452a [DOI] [PubMed] [Google Scholar]
  • 3.Takebe T, Imai R, Ono S. The Current Status of Drug Discovery and Development as Originated in United States Academia: The Influence of Industrial and Academic Collaboration on Drug Discovery and Development. Clin Transl Sci. 2018;11: 597–606. doi: 10.1111/cts.12577 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Peers IS, Ceuppens PR, Harbron C. In search of preclinical robustness. Nat Rev Drug Discov. 2012;11: 733–734. doi: 10.1038/nrd3849 [DOI] [PubMed] [Google Scholar]
  • 5.Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov. 2011;10: 712–712. doi: 10.1038/nrd3439-c1 [DOI] [PubMed] [Google Scholar]
  • 6.Begley CG, Ellis LM. Raise standards for preclinical cancer research. Nature. 2012;483: 531–533. doi: 10.1038/483531a [DOI] [PubMed] [Google Scholar]
  • 7.Freedman LP, Cockburn IM, Simcoe TS. The Economics of Reproducibility in Preclinical Research. PLOS Biol. 2015;13: e1002165. doi: 10.1371/journal.pbio.1002165 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Ioannidis JPA, Kim BYS, Trounson A. How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation. Nat Biomed Eng. 2018;2: 797–809. doi: 10.1038/s41551-018-0314-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Begley CG, Ioannidis JPA. Reproducibility in science: improving the standard for basic and preclinical research. Circ Res. 2015;116: 116–126. doi: 10.1161/CIRCRESAHA.114.303819 [DOI] [PubMed] [Google Scholar]
  • 10.O’Donoghue SI, Baldi BF, Clark SJ, Darling AE, Hogan JM, Kaur S, et al. Visualization of Biomedical Data. Annu Rev Biomed Data Sci. 2018;1: 275–304. doi: 10.1146/annurev-biodatasci-080917-013424 [DOI] [Google Scholar]
  • 11.OECD. Guidance Document on Good In Vitro Method Practices (GIVIMP). 2018. Available: https://www.oecd-ilibrary.org/content/publication/9789264304796-en
  • 12.Vu LP, Pickering BF, Cheng Y, Zaccara S, Nguyen D, Minuesa G, et al. The N6-methyladenosine (m6A)-forming enzyme METTL3 controls myeloid differentiation of normal hematopoietic and leukemia cells. Nat Med. 2017;23: 1369–1376. doi: 10.1038/nm.4416 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Huang P, Qian X, Chen Y, Yu L, Lin H, Wang L, et al. Metalloporphyrin-Encapsulated Biodegradable Nanosystems for Highly Efficient Magnetic Resonance Imaging-Guided Sonodynamic Cancer Therapy. J Am Chem Soc. 2017;139: 1275–1284. doi: 10.1021/jacs.6b11846 [DOI] [PubMed] [Google Scholar]
  • 14.Su R, Dong L, Li C, Nachtergaele S, Wunderlich M, Qing Y, et al. R-2HG Exhibits Anti-tumor Activity by Targeting FTO/m6A/MYC/CEBPA Signaling. Cell. 2018;172: 90–105.e23. doi: 10.1016/j.cell.2017.11.031 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Riss TL, Moravec RA, Niles AL, Duellman S, Benink HA, Worzella TJ, et al. Cell Viability Assays. In: Markossian S, Grossman A, Brimacombe K, Arkin M, Auld D, Austin CP, et al., editors. Assay Guidance Manual. Bethesda (MD): Eli Lilly & Company and the National Center for Advancing Translational Sciences; 2004. Available: http://www.ncbi.nlm.nih.gov/books/NBK144065/ [Google Scholar]
  • 16.Vistica DT, Skehan P, Scudiero D, Monks A, Pittman A, Boyd MR. Tetrazolium-based assays for cellular viability: a critical examination of selected parameters affecting formazan production. Cancer Res. 1991;51: 2515–2520. [PubMed] [Google Scholar]
  • 17.Berridge MV, Herst PM, Tan AS. Tetrazolium dyes as tools in cell biology: new insights into their cellular reduction. Biotechnol Annu Rev. 2005;11: 127–152. doi: 10.1016/S1387-2656(05)11004-7 [DOI] [PubMed] [Google Scholar]
  • 18.Miller I, Min M, Yang C, Tian C, Gookin S, Carter D, et al. Ki67 is a Graded Rather than a Binary Marker of Proliferation versus Quiescence. Cell Rep. 2018;24: 1105–1112.e5. doi: 10.1016/j.celrep.2018.06.110 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Holman L, Head ML, Lanfear R, Jennions MD. Evidence of Experimental Bias in the Life Sciences: Why We Need Blind Data Recording. PLoS Biol. 2015;13: e1002190. doi: 10.1371/journal.pbio.1002190 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Jost AP-T, Waters JC. Designing a rigorous microscopy experiment: Validating methods and avoiding bias. J Cell Biol. 2019;218: 1452–1466. doi: 10.1083/jcb.201812109 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Hsieh J-H. Accounting Artifacts in High-Throughput Toxicity Assays. Methods Mol Biol Clifton NJ. 2016;1473: 143–152. doi: 10.1007/978-1-4939-6346-1_15 [DOI] [PubMed] [Google Scholar]

Decision Letter 0

Hamidreza Montazeri Aliabadi

18 Jun 2021

PONE-D-21-12647

Evidence provided by high-impact cell culture studies does not support authors' claims

PLOS ONE

Dear Dr. Ôzkaya,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

It is important to assure that you reply to all concerns by the reviewers (especially Reviewers 3 and 4). Please note that the review provided by Reviewer 1 pertains only to ‘statistical aspects’ of the study and so does not comment on the ‘clinical aspects’ [like medical importance, relevance of the study, ‘clinical significance and implication(s)’ of the whole study, etc.] of the manuscript.

Please submit your revised manuscript by 6/26/2021. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Hamidreza Montazeri Aliabadi

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please consider posting the list of studies and their analysis as supporting information. If materials, methods, and protocols are well established, authors may cite articles where those protocols are described in detail, but the submission should include sufficient information to be understood independent of these references (https://journals.plos.org/plosone/s/submission-guidelines#loc-materials-and-methods).

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Important note: This review pertains only to ‘statistical aspects’ of the study and so ‘clinical aspects’ [like medical importance, relevance of the study, ‘clinical significance and implication(s)’ of the whole study, etc.] are to be evaluated [should be assessed] separately/independently. Further please note that any ‘statistical review’ is generally done under the assumption that (such) study specific methodological [as well as execution] issues are perfectly taken care of by the investigator(s). This review is not an exception to that and so does not cover clinical aspects {however, seldom comments are made only if those issues are intimately / scientifically related & intermingle with ‘statistical aspects’ of the study}. Agreed that ‘statistical methods’ are used as just tools here, however, they are vital part of methodology [and so should be given due importance].

COMMENTS: In my opinion, this authors group [now there are only two but may include few more similar minded scientists working in same field] can very-well start developing ‘publishing guidelines’ for such articles/literature {I guess, that will be very useful (cell culture studies will likely support authors’ claims subsequently) and definitely will be a great contribution}. It seems that OECD’s ‘Guidance Document on Good In Vitro Method Practices (GIVIMP)’ is not sufficient. Is not that true? There are only nine articles quoted [included in reference, may be because only these many are available] but are good (very relevant) articles. Congratulations.

For this purpose {developing ‘publishing guidelines’ for such articles/literature }, excellent article quoted as reference-1, [Ioannidis JPA. Why Most Published Research Findings Are False. PLOS Med. 2005;2: e124. 260 doi:10.1371/journal.pmed.0020124] may be useful. The understudied aspect of this quality issue in published work highlighted is “the harmony between the hypotheses and the experimental design” is true for many areas and is appreciable. My only question is (line 52) ‘controls (positive, negative)’ {i.e. what are ‘positive controls and negative controls?}. Please clarify. Little/brief clarification of terms ‘p-hacking and cherry picking’ [like ‘P-hacking’ is the relentless analysis of data with an intent to obtain a statistically significant result, usually to support the researcher’s hypothesis and ‘Cherry-picking’ is the presentation of favourable evidence with the concealment of unfavourable evidence] could be useful for readers, I guess.

Although it is not my ‘Subject Area’, in my opinion, this one it is an excellent article. Nevertheless, I feel that overall the article is little short. Maybe adding few more case studies will do. As pointed out in ‘important note’ above “This review pertains only to ‘statistical aspects’ of the study and so ‘clinical aspects’ should be assessed separately/independently. May think of changing title (may make it more catchy).

Reviewer #2: In general, this is a well-conducted and important paper. The Authors are to be congratulated for their attempt to provide a degree of rigor that is typically lacking in preclinical cell culture studies.

In this study, the Authors examined highly-cited cell culture studies and investigated whether the claims made in each study were supported by sufficient experimental evidence.

To do this, they examined 282 claims asserted by 103 different high-impact articles. They found that many of all claims were not sufficiently supported by the evidence that was presented. They also identified troubling examples of misinterpretation of data.

They concluded there was “an alarming discordance” between the actual experimental findings and the way they were actually described in the manuscript.

To perform their analysis, the investigators searched on the key words “cytotoxicity, viability, cell death, growth inhibition, proliferation, or anti-cancer” and retrieved studies from 2017, 2018. They selected 121 publications, each of which had received at least 65 citations. Reviews and studies that did not include a claim or did not involve cell culture were excluded. This left 103 original studies for assessment. In total, the articles that were analysed were cited more than 9,000 times as of March 2021 (within two to four years of publication), so it is reasonable to assume (as the Authors have done) that despite their deficiencies, these publications have had an impact on the field.

The most useful aspects of this work include:

a) Authors provide a useful definition for each of the terms on which they searched - “cytotoxicity, viability, cell death, growth inhibition, proliferation, or anti-cancer”.

b) In Table 1 they provide a clear statement as to what constitutes sufficient and insufficient evidence for each of the terms that they defined. This is a valuable contribution as many publications do not apply the appropriate level of rigor when attempting to perform and interpret these assays.

c) The Authors also present specific proposals to improve the quality of future studies.

It would be a valuable contribution to preclinical research if the Authors’ proposals were widely adopted. This would certainly contribute to an improvement in the robustness of preclinical cell culture experiments.

Areas for improvement.

Major issues.

1) It would be helpful for the Authors to clarify their intent. Specifically, it is not clear to this Reader whether the Authors are stating that 64% of studies were supported with sufficient evidence, or 64% were not supported by sufficient evidence.

The two lines that appear to be in conflict are:

Line 33: “Our findings revealed that only 64% of all claims were sufficiently supported by evidence”

versus

Line 195: “we considered the evidence …which was asserted by 66 different studies (64%), as insufficient”

2) Two areas that were not specifically addressed, but that could further strengthen the manuscript are:

a) The Authors did not mention that subjective analyses should be performed by blinded investigators. Because it is seldom the case that these type of experiments are performed by blinded researchers, it would be valuable for the Authors to highlight its importance.

b) When considering an effect on cell proliferation or cell death, the Authors should stipulate that cell proliferation or cell death should be expressed on a logarithmic not a linear axis. Clearly cell proliferation is an exponential, not a linear function. Although many papers misrepresent their data by presenting it on a linear axis, where it looks more dramatic, this is not an appropriate way to present the data. Given the focus of this publication, this too would be a valuable addition for the Authors to comment upon.

Minor issues:

Line 162: Blc-2 should be Bcl-2

Reviewer #3: PONE-D-21-12647

The authors take on an important reproducibility issue in high-impact cell culture studies. This is certainly of high importance, and suggestions are made for improvements. Overall, the authors comment on a very important point, and one that is close to my own heart… viability vs cytotoxicity. However, they do not do this in a rigorous manner, and this is more suited for a commentary than any sort of research article or review.

1. Line 54 – is this substantiated by any references, or anecdotal?

2. The introduction and overall manuscript is lacking references

3. Lines 67+ reads more like commentary – also a bit vague. What is being discussed here? The cytotoxicity and cell viability terms?

4. What was the rationale for focusing on 2017 and 2018?

5. The inclusion/exclusion should be detailed more clearly on lines 91+

6. Since care is being taken to describe these cell health variables… viability I find a bit problematic. It is not the number of living cells, it generally represents a reaction that is often considered to have a linear relationship to the number of living cells. These enzymes, such as tetrazolium salt, etc., can be modified by exogenous factors and cannot be explicitly considered to describe living cell number.

7. Line 144 – LDH release is a cell viability assay, not a cytotoxicity assay – given the context of this paper, this is a very important point to classify these assays correctly. There are a variety of contexts described in the literature where cells leak LDH and remain otherwise alive and will respond normally via other cell viability measures. Adipocytes is one example of this, though others have been described. Unless the specific experimental context is being taken into account, it’s hard to see how these sorts of generalities are not going to adversely influence results here.

8. Table 1 – how is DNA content not included in cytotoxicity? It is the gold standard for determining cell number. You could say that it is context dependent, if there are no wash steps in a particular assay, but it cannot just be ignored as a cytotoxicity measure.

9. Given the well-described artifacts and issues with a lot of the cell viability and proliferation assays, as described by Hsieh et al., 2015, 2016, etc., this article seems to ignore many of these issues with the assays they included.

10. There doesn’t seem to be any transparency about how and why individual publications were deemed sufficient or insufficient. If highly cited publications are going to be published as insufficient, a rationale column for the decision-making process is critical to providing clarity for readers.

11. Lines 208+ - cytotoxicity is not a measure of viability, it is a different measure completely.

12. Line 206 – “we had to make assumptions on behalf of the researchers” is not very scientific. In a systematic review context, you can and should reach out to the authors for clarification. Appreciate this is not that, but if you are publishing a paper blasting the authors for having an “insufficient” study, and only rely on your interpretation, that is a problem.

13. Moreover, interpretation of who? Is this a single author performing these assumptions? Is each paper reviewed by both authors independently? A single person evaluating these is not robust – and what sort of testing was done to verify that the authors performed these assumptions in a consistent manner?

14. I feel at this point in the paper that I’m not sure of the overall goal. Are the authors just using the assay reported in the paper and checking which box on a table it fits into? This doesn’t seem like the level of discourse that is particularly worthy of a peer-reviewed publication. A commentary could fit this, but it seems more opinion than a rigorous evaluation of reproducibility – which is, of course, crucially important.

15. The first sentence of the discussion is very vague and alarmist – the only thing examined here was the cell proliferation/toxicity/viability and not any other claims these publications may have made. That is very important to make clear

Reviewer #4: The manuscript “Evidence provided by high-impact cell culture studies does not support authors’ claims” from Ozkaya and Geyik describes the use of various assays to evaluate cell state in published articles. The authors mined the literature for highly cited papers from 2017-2018 that evaluated cell viability, death, proliferation, growth, apoptosis, and cytotoxicity in cell culture systems, and considered whether or not the assay was appropriate and sufficient for the conclusions drawn. A summary of the authors’ ‘Evidence Sufficiency’ meta-analysis is presented in a single figure. They suggest that the incorrect use of assays, or over interpretation of results from specific assay classes are likely factors in the irreproducibility of preclinical studies.

Overall, the goal of the manuscript fits under the ‘Systematic Review and Meta-Analyses’ article category; however, it will require significant revisions and additional review should it be considered for publication. I cannot recommend it be published in its current form.

Please address the following points:

Minor

1. Add a citation to papers that have addressed similar issues – line 56-57.

2. Fig. 1C dehydrogenase activity is always used correctly to assess viability?

3. Fig. 1 C (discussed on lines 219-220) you say data are interpreted correctly in 73 claims, but the figure looks like it should be 48 claims. Please correct.

4. Thorough copy-editing is needed

Major

5. Nomenclature recommendations should not be buried in the Methods. You highlight this as a main conclusion of your work. There is an opportunity to be prescriptive with this work and for it to be a resource for people to turn to when selecting assays and designing experiments – I think this should be focused on much more.

6. Your definition of proliferation rate is presented as a population-level readout “how fast a group of cells divide over time” however it is then contradicted by stating that this rate only applies to surviving cells – an apparent decrease in proliferation rate of a population could occur from a slowing of the doubling time of all cells in a population or from a fractional response in which a subpopulation is killed off, and the surviving cells continue to proliferate at the same rate (or at an altered rate) resulting in a longer apparent doubling time, as you point out. Is your proposed definition of proliferation rate meant to be applied to single cell or population level data? Does it only apply to living cells at endpoint? If the endpoint is simply how fast a group of cells divide over time, then it doesn’t matter if the underlying phenotypes are different, though they should be determined with secondary assays designed specifically to measure cell death and growth rate.

7. I strongly disagree that measuring metabolic activity is sufficient for assaying cell viability. In some cases, these techniques (MTT, CellTiter-Glo, etc.) yield results that correlate linearly with live cell count, but many perturbations directly affect the metabolic activity of cells which can result in critical miss-interpretations of effects. See https://pubmed.ncbi.nlm.nih.gov/31302153/ for clear examples.

8. With respect to defining cell death, I generally agree that this needs to be measured directly, however, if the number of cells are counted at the time of drug addition, for example, then a treatment resulting in net cell loss can only result from some degree of cell death. In general, any time cells are counted, live and dead cells should be distinguished at the bare minimum. See Hafner et al. (https://pubmed.ncbi.nlm.nih.gov/27135972/) for a more nuanced and thorough discussion. Growth rate inhibition metrics, an important addition to the field, are omitted and should be discussed – they are immediately relevant and address several of the same concerns the authors highlight.

9. Apoptosis, please correct typo in “molecular switch responsible in apoptosis” to “molecular switch resulting in apoptosis” and specify whether you’re referring to cleaved caspases and/or MOMP.

10. Please clarify why you don’t consider PI sufficient to assay membrane integrity but you do consider other dyes that are also membrane impermeable to be sufficient (lines 174-177) viability assays.

11. Table 1 as is, is confusing especially for the proliferation and viability entries. It would be much more useful if it was presented by claim and then assay, followed by what each assay actually measures, and what is required for that assay to be sufficient for that claim.

12. Only nine references are provided – there is a lot of literature that has been overlooked on assay choices and pitfalls that should be considered, and incorporated into the introduction and discussion sections.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment

Submitted filename: renamed_ab46f.docx

Click here for additional data file. (14.3KB, docx)
PLoS One. 2022 Feb 22;17(2):e0250754. doi: 10.1371/journal.pone.0250754.r002

Author response to Decision Letter 0

23 Jul 2021

Authors: We would like to thank all reviewers for their constructive criticism.

Reviewer #1

Authors: We would like to thank the reviewer for encouraging comments. There is indeed a lack of a comprehensive guideline for basic cell culture research especially regarding simple assays and how to interpret their results. I hope we can eventually start developing such guideline. We will seek likeminded researchers for this purpose as the reviewer has suggested.

Specific points:

R1: My only question is (line 52) ‘controls (positive, negative)’ {i.e. what are ‘positive controls and negative controls?}. Please clarify. Little/brief clarification of terms ‘p-hacking and cherry picking’ [like ‘P-hacking’ is the relentless analysis of data with an intent to obtain a statistically significant result, usually to support the researcher’s hypothesis and ‘Cherry-picking’ is the presentation of favourable evidence with the concealment of unfavourable evidence] could be useful for readers, I guess.

Authors: An explanation has been added for positive and negative controls (lines 53, 54) as well as p-hacking and cherry picking (lines 56, 57).

R1: May think of changing title (may make it more catchy).

We did try to find a catchier title when we first wrote the manuscript, but we did not like the titles we came up with.

Reviewer #2

Authors: Thank you for the reviewer’s comments and support. We tried to be as transparent and as clear as possible when writing the manuscript. Therefore, we really appreciate that our efforts did not go unnoticed.

Specific points

R2: 1) It would be helpful for the Authors to clarify their intent. Specifically, it is not clear to this Reader whether the Authors are stating that 64% of studies were supported with sufficient evidence, or 64% were not supported by sufficient evidence. The two lines that appear to be in conflict are: Line 33: “Our findings revealed that only 64% of all claims were sufficiently supported by evidence” versus Line 195: “we considered the evidence …which was asserted by 66 different studies (64%), as insufficient”

Authors: Those statements are not in conflict, but we can understand the confusion. The first one was the ratio of sufficient claims (180/282). Second one was the ratio of studies with insufficient evidence (66/103). The number being same is just a coincidence. However, during revision we made minor changes in the analysis considering the feedback from the reviewers altering claim sufficiency rate to 63% (178/281) (line: 33). The number of studies with insufficient evidence stayed the same (line: 207)

2) Two areas that were not specifically addressed, but that could further strengthen the manuscript are: a) The Authors did not mention that subjective analyses should be performed by blinded investigators. Because it is seldom the case that these type of experiments are performed by blinded researchers, it would be valuable for the Authors to highlight its importance.

b) When considering an effect on cell proliferation or cell death, the Authors should stipulate that cell proliferation or cell death should be expressed on a logarithmic not a linear axis. Clearly cell proliferation is an exponential, not a linear function. Although many papers misrepresent their data by presenting it on a linear axis, where it looks more dramatic, this is not an appropriate way to present the data. Given the focus of this publication, this too would be a valuable addition for the Authors to comment upon.

Authors: We added a section containing relevant information to the discussion (lines: 280+). We included the importance of blind investigators as well as the importance of logarithmic scale for proliferation studies.

R2: Minor issues: Line 162: Blc-2 should be Bcl-2

Authors: Corrected.

Reviewer #3

Authors: Thank you for your valuable suggestions. We hope that the article is now more rigorous after this revision.

Specific points:

R3: 1. Line 54 – is this substantiated by any references, or anecdotal?

Authors: The references are listed in the previous sentence. Especially by the work of Ioannidis and his colleagues. We originally did not want to repeat the same references in two consecutive sentences. However, in accordance with the comments, we added them to the second sentence as well.

R3: 2. The introduction and overall manuscript is lacking references

Authors. Manuscript was revised and new references were added.

R3: 3. Lines 67+ reads more like commentary – also a bit vague. What is being discussed here? The cytotoxicity and cell viability terms?

Authors: The idea of this study emerged thanks to one simple observation: there were studies using tetrazolium reduction assay results to justify different claims such as viability, cytotoxicity etc. The core of this work is about whether these “claims” are properly sported by “evidence”. Therefore, we tried to clearly define terms corresponding to different claims. For this purpose, we first searched the literature, utilized the OECD guide and finally wrote our own definition. We believe it is important for our readers to understand our process clearly and fully. That is why that section was added.

R3: 4. What was the rationale for focusing on 2017 and 2018?

Authors: We began our analysis in March 2020. We wanted to include a two-year period, we wanted to give the articles one year to receive citations (therefore we did not investigate 2018-2019) and we wanted to focus on the most recent work.

R3: 5. The inclusion/exclusion should be detailed more clearly on lines 91+

Authors: We re-wrote the part.

R3: 6. Since care is being taken to describe these cell health variables… viability I find a bit problematic. It is not the number of living cells, it generally represents a reaction that is often considered to have a linear relationship to the number of living cells. These enzymes, such as tetrazolium salt, etc., can be modified by exogenous factors and cannot be explicitly considered to describe living cell number.

Authors: We agree with the reviewer on the issue. Metabolic activity is not a very good indicator of cell viability. This is especially problematic when your treatment also modifies cellular metabolism. Imagine that you have developed a drug targeting mitochondrial metabolism. It is clear that you should not use tetrazolium reduction assay to evaluate cell viability. However, there are many studies doing exactly that. That is why it is important to address these issues. The same thing can be said for membrane integrity being an indicator of cell death, caspase-3 activation an indicator of apoptosis etc. However, in this study we did not want to focus on that because: even though tetrazolium reduction assay may not be an accurate indicator of viability, it is most definitely not an indicator of apoptosis. Yet there are highly-cited articles claiming just that. They say that the treatment induced apoptosis by providing only MTT results as evidence. This was the clearer and the bigger problem and we focused on that. However, we do take the comments of the reviewers into account and therefore we added a section to the discussion focusing on the problems pointed out (lines: 280+).

R3: 7. Line 144 – LDH release is a cell viability assay, not a cytotoxicity assay – given the context of this paper, this is a very important point to classify these assays correctly. There are a variety of contexts described in the literature where cells leak LDH and remain otherwise alive and will respond normally via other cell viability measures. Adipocytes is one example of this, though others have been described. Unless the specific experimental context is being taken into account, it’s hard to see how these sorts of generalities are not going to adversely influence results here.

Authors: Literature may seem to disagree with the reviewer on this (1-3). LDH assay almost always has been used to measure cytotoxicity. However, we agree that we should question validity of using membrane integrity as a measure of cell death.

References:

1. An enzyme-release assay for natural cytotoxicity. C Korzeniewski, D M Callewaert. 1983

2. Validation of an LDH assay for assessing nanoparticle toxicity. Xianglu Han et al. 2011.

3. Cytotoxicity Assays: In Vitro Methods to Measure Dead Cells. Terry Riss et al. 2019

R3: 8. Table 1 – how is DNA content not included in cytotoxicity? It is the gold standard for determining cell number. You could say that it is context dependent, if there are no wash steps in a particular assay, but it cannot just be ignored as a cytotoxicity measure.

Authors: Table 1 was re-constructed. We did not form the table from out understanding of these assays and the evidence they provide. We formed the table from the data we obtained from the studies we have investigated. Therefore, if there is no study utilizing DNA content as a measure of cytotoxicity we cannot have it there. It is not a general statement; it is just a re-organization of the data we obtained from our analyses. However, since table was criticized for being not clear enough, we created another table and include a more detailed explanation to how we have analyzed the studies and created the table.

R3: 9. 9. Given the well-described artifacts and issues with a lot of the cell viability and proliferation assays, as described by Hsieh et al., 2015, 2016, etc., this article seems to ignore many of these issues with the assays they included.

Authors: We included one of the studies as the reviewer suggested.

R3: 10. There doesn’t seem to be any transparency about how and why individual publications were deemed sufficient or insufficient. If highly cited publications are going to be published as insufficient, a rationale column for the decision-making process is critical to providing clarity for readers.

Authors: We re-constructed table 1 to draw a clearer picture of our evaluation process. We included a rationale column as the reviewer suggested.

R3: 11. Lines 208+ - cytotoxicity is not a measure of viability, it is a different measure completely.

Authors: We completely agree on cytotoxicity not being a measure of viability. However, we encountered articles using cytotoxicity as if it represents viability during our investigation. We included that section to acknowledge it. We do not defend it.

R3: 12. Line 206 – “we had to make assumptions on behalf of the researchers” is not very scientific. In a systematic review context, you can and should reach out to the authors for clarification. Appreciate this is not that, but if you are publishing a paper blasting the authors for having an “insufficient” study, and only rely on your interpretation, that is a problem.

R3: 13. Moreover, interpretation of who? Is this a single author performing these assumptions? Is each paper reviewed by both authors independently? A single person evaluating these is not robust – and what sort of testing was done to verify that the authors performed these assumptions in a consistent manner?

Authors: There is no assumption and interpretation in the analysis part. The assumptions /interpretations were general and were carried out them before the analysis. The assumptions are listed as definitions in the materials and methods section. When we define viability, cytotoxicity, proliferation etc. we “assume” they the authors of the articles we are analyzing have used it the same way. But since we cannot know it for sure we included the section previously discussed (lines 217+)

We agree that out evaluation process was not clear enough in the original manuscript. We re-constructed Table 1 and included more text to clarify our process.

R3: 14. I feel at this point in the paper that I’m not sure of the overall goal. Are the authors just using the assay reported in the paper and checking which box on a table it fits into? This doesn’t seem like the level of discourse that is particularly worthy of a peer-reviewed publication. A commentary could fit this, but it seems more opinion than a rigorous evaluation of reproducibility – which is, of course, crucially important.

Authors: We included more text to clarify our process. We tried out best to make the analysis as objective as possible. This study is not an evaluation of reproducibility. We focused on the claims and the evidence of the articles we have investigated. This study is neither a commentary nor a review. We had a hypothesis “highly cited articles do not have sufficient evidence to support their cell culture-related claims” and we focused on selected claims developed a strategy to test the hypothesis. We gathered data, made our analysis and have reached a conclusion.

R3: 15. The first sentence of the discussion is very vague and alarmist – the only thing examined here was the cell proliferation/toxicity/viability and not any other claims these publications may have made. That is very important to make clear

Authors: The discussion was revised.

Reviewer #3

Authors: We would like to thank the reviewer for helpful comments that substantially improved the quality of this manuscript. We hope that it could be considered for publication at its current stage.

R4: Add a citation to papers that have addressed similar issues – line 56-57.

Authors: Citations added.

R4: Fig. 1C dehydrogenase activity is always used correctly to assess viability?

Authors: That is correct. In this study we considered dehydrogenase activity measurement to provide sufficient evidence for viability independent from the experimental design. On the other hand, we are well aware of metabolic activity not being a very accurate indicator of cell viability. We included a section in discussion to discuss it (lines 280+).

R4: 3. Fig. 1 C (discussed on lines 219-220) you say data are interpreted correctly in 73 claims, but the figure looks like it should be 48 claims. Please correct.

Authors: Corrected.

R4: 4. Thorough copy-editing is needed

Authors: Manuscript was revised accordingly.

R4: 5. Nomenclature recommendations should not be buried in the Methods. You highlight this as a main conclusion of your work. There is an opportunity to be prescriptive with this work and for it to be a resource for people to turn to when selecting assays and designing experiments – I think this should be focused on much more.

Authors: That was our original intention. However, we realized that we needed those definitions in the materials and methods section to clearly explain our process. Considering the reviewers’ comments, we also included the definitions to discussion section and referred to them in the abstract.

R4: 6. Your definition of proliferation rate is presented as a population-level readout “how fast a group of cells divide over time” however it is then contradicted by stating that this rate only applies to surviving cells – an apparent decrease in proliferation rate of a population could occur from a slowing of the doubling time of all cells in a population or from a fractional response in which a subpopulation is killed off, and the surviving cells continue to proliferate at the same rate (or at an altered rate) resulting in a longer apparent doubling time, as you point out. Is your proposed definition of proliferation rate meant to be applied to single cell or population level data? Does it only apply to living cells at endpoint? If the endpoint is simply how fast a group of cells divide over time, then it doesn’t matter if the underlying phenotypes are different, though they should be determined with secondary assays designed specifically to measure cell death and growth rate.

Authors: The main point is to differentiate slowing of the doubling time from killing a sub-population. We embraced that distinction to differentiate “proliferation rate” from “viability” and to give it a unique identity. We agree that the best way to measure it is to measure growth rate along with the cell death and we debated to decide what would be an acceptable evidence to demonstrate proliferation rate independent from cell death. Beside the usual markers which were specifically designed to measure proliferation rate, we decided “if there is no apparent reason for cell death (i.e. no treatment) and cells are being observed over time to observe how fast a group of cells divide over time, viability change can indicate proliferation”. However, it is important to note that while measuring proliferation-specific one must also consider the impact of cell loss due to death similar to the case with the viability. In essence, if cells die via cell death you may end up reading lower levels of Ki67 expression or nucleotide incorporation. That is why the proliferation data should also be normalized based on living cells. We included that recommendation to the discussion (lines: 280+).

R4: 7. I strongly disagree that measuring metabolic activity is sufficient for assaying cell viability. In some cases, these techniques (MTT, CellTiter-Glo, etc.) yield results that correlate linearly with live cell count, but many perturbations directly affect the metabolic activity of cells which can result in critical miss-interpretations of effects. See https://pubmed.ncbi.nlm.nih.gov/31302153/ for clear examples.

We agree with the reviewer. We do not know how accurately metabolic activity indicates cell viability. This is especially problematic when your treatment also modifies cellular metabolism. Imagine that you have developed a drug targeting mitochondrial metabolism. It is clear that you should not use tetrazolium reduction assay to evaluate cell viability. However, there are many studies doing exactly that. That is why it is important to address these issues. The same thing can be said for membrane integrity being an indicator of cell death, caspase-3 activation an indicator of apoptosis etc. However, in this study we did not want to focus on that because: even though tetrazolium reduction assay may not be an accurate indicator of viability, it is most definitely not an indicator of apoptosis. Yet there are highly-cited articles claiming just that. They say that the treatment induced apoptosis by providing only MTT results as evidence. This was the clearer and the bigger problem and we focused on that. However, we do take the comments of the reviewers into account and therefore we added a section to the discussion focusing on the problems pointed out (lines: 280+).

R4: 8. With respect to defining cell death, I generally agree that this needs to be measured directly, however, if the number of cells are counted at the time of drug addition, for example, then a treatment resulting in net cell loss can only result from some degree of cell death. In general, any time cells are counted, live and dead cells should be distinguished at the bare minimum. See Hafner et al. (https://pubmed.ncbi.nlm.nih.gov/27135972/) for a more nuanced and thorough discussion. Growth rate inhibition metrics, an important addition to the field, are omitted and should be discussed – they are immediately relevant and address several of the same concerns the authors highlight.

Authors: We included discussion of these metrics (lines 270+).

R4: 9. Apoptosis, please correct typo in “molecular switch responsible in apoptosis” to “molecular switch resulting in apoptosis” and specify whether you’re referring to cleaved caspases and/or MOMP.

Authors: Corrected

R4: 10. Please clarify why you don’t consider PI sufficient to assay membrane integrity but you do consider other dyes that are also membrane impermeable to be sufficient (lines 174-177) viability assays.

Authors: Thanks to the comments of the reviewer, we realized that we made a mistake evaluating the use of membrane integrity markers for measurement of viability. We now classify PI and AAD7 as insufficient. We only accepted trypan blue as sufficient because living cells can be detected via trypan blue assay. We re-calculated all the numbers and re-draw all the graphs accordingly.

R4: 11. Table 1 as is, is confusing especially for the proliferation and viability entries. It would be much more useful if it was presented by claim and then assay, followed by what each assay actually measures, and what is required for that assay to be sufficient for that claim.

Authors: Table-1 is reconstructed to better reflect our evaluation process.

R4: 12. Only nine references are provided – there is a lot of literature that has been overlooked on assay choices and pitfalls that should be considered, and incorporated into the introduction and discussion sections.

Authors: More references are added.

Attachment

Submitted filename: Ozkaya Response to Reviewers.docx

Click here for additional data file. (27KB, docx)

Decision Letter 1

Hamidreza Montazeri Aliabadi

19 Aug 2021

PONE-D-21-12647R1

Evidence provided by high-impact cell culture studies does not support authors' claims

PLOS ONE

Dear Dr. Özkaya,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please specially address the concerns raised by reviewers 3 and 4. It might be helpful, if you can revise the title of the manuscript, so it is more specific ad does not seem dismissive towards all "authors' claims". Also, the text could also be slightly revised to clarify that the objective is not to dismiss all the findings these publications are reporting.

Please submit your revised manuscript by 9/10/2021. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Hamidreza Montazeri Aliabadi

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: (No Response)

Reviewer #4: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

Reviewer #4: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: N/A

Reviewer #4: (No Response)

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: (No Response)

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: COMMENTS: As already said, that everything [like execution, methodology, statistical analyses, etc] are very good [i.e. study is excellent]. Writing/presentation is also excellent. Later, comments made [probably by other respected reviewers as well] were/are attended or answered positively/adequately, I am fully satisfied and the manuscript is improved a lot. I recommend acceptance.

Reviewer #2: Thank you for addressing the comments that I had provided.

This is a useful contribution and an important contribution.

Reviewer #3: I appreciate the reviewer edits to the manuscript, though many of my original concerns remain. This article still reads to me as a commentary - an important topic for one - but is not well enough defined for publication as a research article.

I still have major concerns with the title of this article. The authors rather blindly call out other papers yet (Evidence provided by high-impact cell culture studies does not support authors' claims) yet they have not examined anything in these papers other than the use of cytotoxicity/cell viability assays. This absolutely must be made crystal clear. I asked for this on first revision and this was not addressed. The inflammatory title, when this is explicitly not assessed, is not proper.

It also needs to be explicitly stated that the claims they are examining from these papers are in some cases only a very small part of the overall paper. The authors in the response to reviewer document repeatedly state the rationale for this article, but they have strayed far beyond that in the paper itself, as written.

I happen to disagree with the authors on the LDH assay, though appreciate there can be difference of opinion here. Though the irony of them citing the literature to defend correct interpretation is not lost on me, given the topic of their paper. I would suggest that they think about what this assay actually examines and whether that fits into the definition of a cell viability or toxicity assay. As there are clear examples where the cell type and context promote positive findings in this assay, it cannot be a cytotoxicity assay.

Some language editing is still needed here, particularly for the newly added sections highlighted in yellow.

This article should be a much shorter commentary, where it could accomplish the author's goals and not over-reach.

Again, how is DNA content listed under proliferation and not toxicity? This was not appropriately responded to in revision.

It is still unclear how some terms are being distinguished. On line 135, cytotoxicity, the definition specifically states "cell death". Yet in Table 1, cytotoxicity and cell death are separate groups on this table. I don't understand how these groups are being defined and how they are different, given the descriptions the authors are providing here.

Reviewer #4: PONE-D-21-12647_R1 review (also attached)

The revised manuscript “Evidence provided by high-impact cell culture studies does not support authors’ claims” from Ozkaya and Geyik is improved from the initial submission; however, several points still need to be addressed and clarified prior to publication. The expansion of Table 1 is a significant improvement to the previous version.

Minor points:

1. Table 1: I think it would be more intuitive to change the order slightly so the cell death claims (apoptosis, cell death, cytotoxicity) are followed by the live cell claims together (cell growth, proliferation, viability) rather than strictly listing them alphabetically.

2. Thorough copy editing is still required. Much of the text remains unchanged.

3. Referring to cell state assays as ‘cell culture techniques’ (for example on lines 41, 266 and 282) is confusing. I perceive good cell culture technique to be the proper maintenance of cells in culture (free of mycoplasma and other contaminants, log-phase growth for the assays you discuss etc). Good cell culture technique is also a critical factor in generating high quality and reproducible measures of cell viability/death/proliferation etc. It would be worth adding that all of the assays discussed can be badly confounded by poor cell culture technique. As you correctly point out, cost, simplicity, speed, and throughput likely motivate assay selection.

4. Reference to Fig 1D (line 226) should be to 1C.

5. While a valuable clinical biomarker, ki67 positivity as it relates to cell cycle and cells cycling is complicated in culture systems, and many cells in arrested states remain ki67 positive – caution in its interpretation is warranted, and like most of the endpoints discussed should be cross-validated. See https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6108547/ and https://elifesciences.org/articles/13722

6. Lines 37-38 are unclear.

7. Consider reframing line 76 from ‘we decided to define these terms ourselves’ to ‘in the current work, we propose a series of definitions and recommendations’ since this is the ultimate goal.

8. Lines 132 and 133, please be specific about the ‘morphological changes’ and ‘molecular switches’ that you are referring to.

9. Please add the citations for id#9 and id#26 in the text (lines 260 and 261) so the reader can refer to them directly without having to access your web tool.

10. Line 271: GR refers to normalized growth rate inhibition, not simply growth rate.

11. Line 272: DIP should be specified ‘drug-induced proliferation rate’

12. Lines 273-276 are unclear. Please explain.

Major points:

1. Your claim in Table 1 that ‘DH activity correlates with the number of viable cells’ is not always true; at the very least, the associated ‘Yes’ should be changed to ‘Conditional’. Moreover, you twice assert that you consider measurements of metabolic activity as sufficient evidence for viability because they’re widely used: ‘…it is the most common method used to evaluate viability…’ (line 119) and ‘…we considered measurement of metabolic activity as an indicator of viability as it is by far the most preferred viability assay…’ (lines 283-284). This is poor justification for reasons you include in the discussion and reasons brought up in the previous reviews. With proper controls/assay calibration (how does the assay output relate to live cell count in the specific context?) and cross-validation experiments to confirm that metabolic activity is not influenced by the perturbation under study, these assays of metabolic activity can be used as surrogates for viability, but these controls are critical to correct data interpretation. My concern is that these points are not sufficiently addressed, and that incorrect use of metabolic assays to evaluate cell viability and proliferation is presented as acceptable without caveats throughout the body of the paper only to be contradicted in the discussion.

2. Can you comment on how often, in the papers you included, the assays you considered are the only line of evidence for a conclusion drawn since this seems to be the most problematic case? A common scenario is one where a crude, population level measure such as MTT is used to suggest toxicity, and then followed by a microscopy or flow cytometry-based assay to confirm cell death vs change in proliferation and mechanism (apoptosis vs necrosis etc). In a case like that, would the use of a metabolic assay be considered sufficient or insufficient? The use of complimentary and confirmatory assays should be discussed and considered when scoring claims made in the papers that you evaluated.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr. Sanjeev Sarmukaddam

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment

Submitted filename: PLOS ONE R1 review.docx

Click here for additional data file. (15.8KB, docx)
PLoS One. 2022 Feb 22;17(2):e0250754. doi: 10.1371/journal.pone.0250754.r004

Author response to Decision Letter 1

23 Sep 2021

Please specially address the concerns raised by reviewers 3 and 4. It might be helpful, if you can revise the title of the manuscript, so it is more specific ad does not seem dismissive towards all "authors' claims". Also, the text could also be slightly revised to clarify that the objective is not to dismiss all the findings these publications are reporting.

The title was changed, and we included several statements to different sections of the article to clarify the points the editor made.

Evidence provided by high-impact cell culture studies does not support authors’ claims

Reviewer #1: COMMENTS: As already said, that everything [like execution, methodology, statistical analyses, etc] are very good [i.e. study is excellent]. Writing/presentation is also excellent. Later, comments made [probably by other respected reviewers as well] were/are attended or answered positively/adequately, I am fully satisfied and the manuscript is improved a lot. I recommend acceptance.

Thank you.

Reviewer #2: Thank you for addressing the comments that I had provided.

This is a useful contribution and an important contribution.

Thank you.

Reviewer #3: I appreciate the reviewer edits to the manuscript, though many of my original concerns remain. This article still reads to me as a commentary - an important topic for one - but is not well enough defined for publication as a research article.

We believe that this work fits the definition of a “research article” embraced by PlosOne:

“Research Articles present the results of original research that address a clearly defined research question and contribute to the body of academic knowledge”

We have a clearly defined research question (are cell culture claims of high-impact publications sufficiently supported by experimental evidence), we have an experimental approach to test the hypothesis and obtained results. Finally, we believe that the findings of this study as well as our conclusions contribute to the body of academic knowledge.

We leave the final decision on the matter to the editor.

I still have major concerns with the title of this article. The authors rather blindly call out other papers yet (Evidence provided by high-impact cell culture studies does not support authors' claims) yet they have not examined anything in these papers other than the use of cytotoxicity/cell viability assays. This absolutely must be made crystal clear. I asked for this on first revision and this was not addressed. The inflammatory title, when this is explicitly not assessed, is not proper.

We have changed the title.

It also needs to be explicitly stated that the claims they are examining from these papers are in some cases only a very small part of the overall paper. The authors in the response to reviewer document repeatedly state the rationale for this article, but they have strayed far beyond that in the paper itself, as written.

We have changed the title and included statements to the abstract, introduction, and discussion sections to clarify this.

I happen to disagree with the authors on the LDH assay, though appreciate there can be difference of opinion here. Though the irony of them citing the literature to defend correct interpretation is not lost on me, given the topic of their paper. I would suggest that they think about what this assay actually examines and whether that fits into the definition of a cell viability or toxicity assay. As there are clear examples where the cell type and context promote positive findings in this assay, it cannot be a cytotoxicity assay.

After discussing the points made by the reviewer, we decided not to change our original position on the matter. Cytotoxic insult causes loss of membrane integrity and intracellular content such as LDH leak from death/dying cells. Measuring LDH activity in cell media correlates with the loss of membrane integrity thus, cytotoxicity. Like every indirect assay, it has its limitations, but those limitations are not enough to dismiss it.

It is important to note that the literature we cited in our previous response were carefully selected as they explain why membrane integrity was an indicator of cell death providing experimental evidence. We want to make it clear that even if we criticize published work in our study, we still believe citing published work is one of the pillars of science.

Some language editing is still needed here, particularly for the newly added sections highlighted in yellow.

Proofreading on manuscript was carried out by a native English speaker and changes were made accordingly.

This article should be a much shorter commentary, where it could accomplish the author's goals and not over-reach.

This has been answered in first point.

Again, how is DNA content listed under proliferation and not toxicity? This was not appropriately responded to in revision.

DNA content is listed under proliferation because that is how the articles we have investigated have used them. We did not decide which evidence supports which claims. That information was taken from the articles we have investigated. Since there was no article using DNA content to measure toxicity, we cannot have it in that list.

In other words, we can summarize the methodology used as:

1- Find the studies that contains at least one claim (such as cytotoxicity, proliferation, etc.)

2- Look for the methods used to make “that” claim in those articles (such as DNA content, dehydrogenase activity, etc.)

3- Decide whether the method used by the article provides sufficient evidence for the claim asserted in the article.

For example, when Study A claims cytotoxicity by using MTT, we included that to our analysis and decided whether it is sufficient or not.

Another example, when Study B claims proliferation by using DNA content, we included that to analysis and decided whether it is sufficient or not.

So, if any study among the investigated 103 studies have had claimed cytotoxicity by demonstrating DNA amount as evidence, we would have included that. And then we would have decided the sufficiency. However, there were no studies using DNA content as evidence for cytotoxicity.

By the way, we also agree with the reviewer on DNA content measurement being an appropriate technique to detect cytotoxicity, it just was not used in any of the 103 studies we analyzed and that is why it was not on the table.

It is still unclear how some terms are being distinguished. On line 135, cytotoxicity, the definition specifically states "cell death". Yet in Table 1, cytotoxicity and cell death are separate groups on this table. I don't understand how these groups are being defined and how they are different, given the descriptions the authors are providing here.

The table was generated by using the information gathered from the articles we have investigated. So, if the authors of an article used the term “cell death” or “cytotoxicity” in their manuscript, we included them separately in the table. Therefore, different claims in the table do not correspond to different definitions, that is just the data obtained from the articles. In fact, we considered cytotoxicity and cell death the same in our analysis. Similarly, we also considered growth and proliferation the same as well. However, we have written them separately in the table because the table contains the claims asserted by the authors of the articles we have investigated, and we just copied whatever they have chosen to write in their manuscripts. On line 135, on the other hand, we have written the definitions we have embraced in this study. That is why it specifically states “cell death” under cytotoxicity. We consider them as being equal and we carried out our analysis accordingly.

Reviewer #4: PONE-D-21-12647_R1 review (also attached)

The revised manuscript “Evidence provided by high-impact cell culture studies does not support authors’ claims” from Ozkaya and Geyik is improved from the initial submission; however, several points still need to be addressed and clarified prior to publication. The expansion of Table 1 is a significant improvement to the previous version.

Minor points:

1. Table 1: I think it would be more intuitive to change the order slightly so the cell death claims (apoptosis, cell death, cytotoxicity) are followed by the live cell claims together (cell growth, proliferation, viability) rather than strictly listing them alphabetically.

We have changed the order as recommended.

2. Thorough copy editing is still required. Much of the text remains unchanged.

Proofreading on manuscript was carried out by a native English speaker and changes were made accordingly.

3. Referring to cell state assays as ‘cell culture techniques’ (for example on lines 41, 266 and 282) is confusing. I perceive good cell culture technique to be the proper maintenance of cells in culture (free of mycoplasma and other contaminants, log-phase growth for the assays you discuss etc). Good cell culture technique is also a critical factor in generating high quality and reproducible measures of cell viability/death/proliferation etc. It would be worth adding that all of the assays discussed can be badly confounded by poor cell culture technique. As you correctly point out, cost, simplicity, speed, and throughput likely motivate assay selection.

Word “techniques” in mentioned lines was changed to “methods”.

4. Reference to Fig 1D (line 226) should be to 1C.

Thank you for the nice catch. We have corrected the figure reference.

5. While a valuable clinical biomarker, ki67 positivity as it relates to cell cycle and cells cycling is complicated in culture systems, and many cells in arrested states remain ki67 positive – caution in its interpretation is warranted, and like most of the endpoints discussed should be cross-validated. See https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6108547/ and https://elifesciences.org/articles/13722

A part was added to the discussion to mention this.

6. Lines 37-38 are unclear.

The part was re-written.

7. Consider reframing line 76 from ‘we decided to define these terms ourselves’ to ‘in the current work, we propose a series of definitions and recommendations’ since this is the ultimate goal.

We have reframed the sentence as recommended.

8. Lines 132 and 133, please be specific about the ‘morphological changes’ and ‘molecular switches’ that you are referring to.

The part was specified.

9. Please add the citations for id#9 and id#26 in the text (lines 260 and 261) so the reader can refer to them directly without having to access your web tool.

Citations were added.

10. Line 271: GR refers to normalized growth rate inhibition, not simply growth rate.

It was corrected.

11. Line 272: DIP should be specified ‘drug-induced proliferation rate’

It was corrected.

12. Lines 273-276 are unclear. Please explain.

The part was re-written.

Major points:

1. Your claim in Table 1 that ‘DH activity correlates with the number of viable cells’ is not always true; at the very least, the associated ‘Yes’ should be changed to ‘Conditional’. Moreover, you twice assert that you consider measurements of metabolic activity as sufficient evidence for viability because they’re widely used: ‘…it is the most common method used to evaluate viability…’ (line 119) and ‘…we considered measurement of metabolic activity as an indicator of viability as it is by far the most preferred viability assay…’ (lines 283-284). This is poor justification for reasons you include in the discussion and reasons brought up in the previous reviews. With proper controls/assay calibration (how does the assay output relate to live cell count in the specific context?) and cross-validation experiments to confirm that metabolic activity is not influenced by the perturbation under study, these assays of metabolic activity can be used as surrogates for viability, but these controls are critical to correct data interpretation. My concern is that these points are not sufficiently addressed, and that incorrect use of metabolic assays to evaluate cell viability and proliferation is presented as acceptable without caveats throughout the body of the paper only to be contradicted in the discussion.

We agree with the reviewer, and we have made minor revisions to emphasize these points. However, after many discussions we decided not to change our analysis. Every assay has its limitations, and we must draw the line somewhere. In this study we decided to consider metabolic activity as an acceptable indicator of cell viability. The reviewer is correct on “being widely used” to be a poor justification. However, it is not only about being a commonly used tool. Viability measurement is the only purpose of an MTT assay that is somewhat acceptable. Again, it may not be perfect to measure viability, but the problem is that people are using it to measure cell death which is arguably far worse than using it to measure viability. And that is what we wanted to focus on this study.

We investigated randomly selected articles from our list to detect whether they have used controls to confirm effects on metabolism. None of the articles have done a proper assay calibration. Therefore, when we change the sufficiency of MTT from “yes” to “conditional”, it is instead going to be “no” for almost all the claims. We are simply not ready to consider MTT as insufficient evidence for viability.

Another issue with that approach was about the way this article presents its findings. We wanted to keep it as simple as possible to reach a wider audience. When we dive into the detail level in which we consider MTT as conditional evidence for viability, almost all experimental results are going to be “conditional evidence”. Indeed, proper controls and normalization are important for every assay, but that approach would turn this work into an article about cell culture assays and their weaknesses as we won’t be able to classify them as sufficient/insufficient for specific claims. We know that our current approach is also reductionist, but the idea is being not as reductionist as the work we criticize. By doing that we are hoping to reach more scientists and if we can make a couple of researchers question their decision to claim cell death with MTT data, we consider it as a win. After that initial step we can move on to a more complicated approach. We hope that someday we can be a part of team which creates a comprehensive guideline (on viability/cytotoxicity/proliferation rate etc) by which the current methodology is dissected, evaluated and practical recommendations are made.

2. Can you comment on how often, in the papers you included, the assays you considered are the only line of evidence for a conclusion drawn since this seems to be the most problematic case? A common scenario is one where a crude, population level measure such as MTT is used to suggest toxicity, and then followed by a microscopy or flow cytometry-based assay to confirm cell death vs change in proliferation and mechanism (apoptosis vs necrosis etc). In a case like that, would the use of a metabolic assay be considered sufficient or insufficient? The use of complimentary and confirmatory assays should be discussed and considered when scoring claims made in the papers that you evaluated.

When we were evaluating the articles’ claims we focused on the strongest evidence. So, if the article claimed apoptosis and has MTT and Annexin-V as evidence, we did not list MTT for their apoptosis claim, we just listed Annexin-V and considered it sufficient. That approached kept our list simple and concise. We detailed our methodology it a bit more in materials and methods section to clarify this.

However, if the claim specifically states MTT without mentioning further experimentation we focused on that. For example: article no4 claimed proliferation rate change and performed an MTT and nucleotide incorporation assay. However, since there is this statement “MTT assay was performed to assess cell proliferation” we included it in our list as a separate entry and considered it insufficient.

Article no121 claimed viability, cytotoxicity, cell death and apoptosis. They performed MTT and Annexin-V assays. However, since they state “The cytotoxicity of mPEG–PCL micelle to MCF-7 cells was evaluated using MTT assays.” we included MTT as a separate entry in the list and considered it as insufficient.

Like this examples in all of the cases we did not focus on what really happened to the cells in the experiments, instead we focused on how it was presented/discussed in the article.

On the other hand, many articles were easier to analyze including article no9 claiming viability, cytotoxicity, apoptosis and proliferation rate inhibition. The only data they had: MTT.

We rechecked articles with multiple evidence to detect any cases in which authors provide a better follow up evidence that we might have missed. During that investigation we decided that the article no26 has stronger evidence for anti-cell activity (anti leukemic cell activity, in their case). That evidence was in tissue level and that is why we have missed it in the first place. Consequently, it was removed from the claim list. We reanalyzed the data based on this alteration and changed relevant parts accordingly.

Attachment

Submitted filename: Ozkaya 2nd Response to Reviewers.docx

Click here for additional data file. (25KB, docx)

Decision Letter 2

Hamidreza Montazeri Aliabadi

28 Oct 2021

PONE-D-21-12647R2From viability to cell death: claims with insufficient evidence in high-impact cell culture studiesPLOS ONE

Dear Dr. Ozkaya,

I would like to start by apologizing for the lengthy review process. Your manuscript, as I am sure you know, is about a controversial subject that has triggered discussions among the reviewers. As you will see below, all reviewers agree that the latest version of the manuscript has improved significantly; however, while reviewers 1 and 2 both agreed that manuscript should be accepted, two reviewers (Reviewers 3 and 4), have cited a few minor points that requires your attention. Hopefully, they are not too time-consuming to answer, and this would be the final re-submission.

Please submit your revised manuscript by 11/4/2021. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Hamidreza Montazeri Aliabadi

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: (No Response)

Reviewer #4: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: (No Response)

Reviewer #2: Yes

Reviewer #3: Partly

Reviewer #4: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: (No Response)

Reviewer #2: Yes

Reviewer #3: N/A

Reviewer #4: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: (No Response)

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: (No Response)

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: COMMENTS: As already said, methodology, statistical analyses, etc are good and the manuscript is improved a lot. I recommend acceptance.

Reviewer #2: No additional comments - my previous comments have been adequately addressed.

Congratulations on a valuable study.

Reviewer #3: The authors have done a good job of responding to most reviewer concerns, though on re-review I have a few limited concerns. The bulk of the data from this study is provided in an alternate web link, though I'm not sure how I feel about this. Is this a stable link? Is the journal providing certainty that this will stay active in perpetuity? This paper becomes completely illegible without these supporting data. I strongly suggest they be added as supplemental data to the manuscript itself to ensure that these data are retained.

I still feel there are some lingering issues, including some raised by reviewer #4. The final question about multiple assays, I think, raises some concerns with this analysis. This addresses how articles using multiple assays or measures were treated. The author answer was that they judged the best measure and judged the article on that - though insufficient details are provided on how they judged the best assay for each specific paper and condition. Also, certainly some credit should be given to authors that couple multiple assays together - these can often provide a more complete picture of cell health than any one individual assay.

I also feel like there are some interesting concerns raised by reviewer 4 in regard to controls and the author answer to it. Surely the use of an assay, without appropriate controls, is not sufficient. Yet it seems that the details of the assay and the specifics of how it was performed and controlled for are not factored into these determinations. When making this still broad statement about the credibility of these author claims, this seems an important aspect to have considered.

Reviewer #4: The article revised by Ozkaya and Geyik is improved. In particular I appreciate the addition of discussion around the caveats of the study and assays discussed. However, I find that the article has gained length rather than clarity, and that the authors are still missing an opportunity to provide clear recommendations for good cell culture technique (as it pertains to all cell-based assays) and optimal assay use to address specific biological questions accurately. While I see value in drawing attention to widespread incorrect or over-interpretation from basic assays, the potential for impact lies in being explicit about what would have been needed for a claim to be deemed sufficient. This information is only included in the Methods and is incomplete (examples are provided), the authors could consider adding a summary table of assays and the conclusions that can be drawn from them. Although I will leave it to the editor to decide whether or not this falls within the scope of the current paper.

I continue to disagree with the authors’ assessment that ‘viability measurement is the only purpose of an MTT assay that is somewhat acceptable’, when really it is evaluating changes in metabolic activity in a population of cells. The same number of cells can have vastly different MTT readouts based on treatment, and while I agree that using these assays to assert cell death is particularly problematic, the oversimplification is detrimental to the paper. An increase in signal does not always indicate an increase in cell number.

Remove references to the discussion from the Methods section.

I do not understand why cell count and DNA content are only considered sufficient evidence for proliferation in the absence of treatment. Background levels of cell death need to be assayed under all conditions, controls included.

Why is ‘Real-time cell imaging’ (which I assume to refer to time-lapse microscopy) insufficient for quantifying cell death? Watching cells over time is both a straightforward and accurate way to monitor cell fate.

DAPI is membrane permeable, how is it correctly used to test membrane integrity? It can be used for DNA content, and typically is used to complement a membrane impermeable dye to accurately evaluate the fraction of cell death. Same concern for acridine orange and Hoechst in the Viability section of the table. Unless they are used with GhostDye (or PI etc), or DNA content is accounted for, they can only yield overall cell counts.

Please clarify the difference between ‘Dye inclusion’ and ‘Membrane integrity’.

Propidium iodine should be Propidium iodide.

pH3 positive cells represent a small fraction of a population and can represent cells arrested in M-phase, their presence alone does not reflect the overall proliferative potential of a population.

Change in proliferation is not limited to decrease in rate, perturbations can also stimulate proliferation.

The discussion needs to be re-worked and copy edited. In particular lines 297-310. The discussion of IC50 values does not seem to fit – although these methods aim to improve reproducibility of large-scale drug response studies, this is somewhat separate from the issues the current paper is trying to address.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr. Sanjeev Sarmukaddam

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2022 Feb 22;17(2):e0250754. doi: 10.1371/journal.pone.0250754.r006

Author response to Decision Letter 2

10 Dec 2021

Reviewer #3: The authors have done a good job of responding to most reviewer concerns, though on re-review I have a few limited concerns. The bulk of the data from this study is provided in an alternate web link, though I'm not sure how I feel about this. Is this a stable link? Is the journal providing certainty that this will stay active in perpetuity? This paper becomes completely illegible without these supporting data. I strongly suggest they be added as supplemental data to the manuscript itself to ensure that these data are retained.

We have created an CSV document (excel file) and upload it as a supplementary data along with the manuscript.

Reviewer #3: I still feel there are some lingering issues, including some raised by reviewer #4. The final question about multiple assays, I think, raises some concerns with this analysis. This addresses how articles using multiple assays or measures were treated. The author answer was that they judged the best measure and judged the article on that - though insufficient details are provided on how they judged the best assay for each specific paper and condition. Also, certainly some credit should be given to authors that couple multiple assays together - these can often provide a more complete picture of cell health than any one individual assay.

When multiple methods are used, and different types of evidence are provided by a study we focused on the best one. Here when we say “best” what we mean is the best according to the table 1. For example, let’s say researchers have performed assays A, B and C to evaluate cytotoxicity. And let’s say, according to the table 1, A and B is “insufficient” while C is “sufficient”. In that case we just included “C” and ignored the use of “A” and “B” as they have more of a supportive role. Basically, if there were multiple types of evidence supporting a claim we only focused on “sufficient” ones, if there are any. A simple example would be apoptotic markers. Bcl and Bax expressions are usually measured along with DNA fragmentation or phosphotidyl-serine exposure. If we were to evaluate those separately, we should classify them as insufficient. But since there is a more relevant and sufficient evidence presented along with them, we exclude them from our analysis. We do agree that utilizing multiple assays would generate better evidence but we think that evaluating that would be out of scope of this work.

Reviewer #3: I also feel like there are some interesting concerns raised by reviewer 4 in regard to controls and the author answer to it. Surely the use of an assay, without appropriate controls, is not sufficient. Yet it seems that the details of the assay and the specifics of how it was performed and controlled for are not factored into these determinations. When making this still broad statement about the credibility of these author claims, this seems an important aspect to have considered.

Although we agree that the proper use of controls is crucial to obtain good evidence, it is again not the focus of this study. This work is planned and conducted to demonstrate the misuse and misinterpretation of some cell culture methods and its high prevalence even in high impact articles. It is not however, designed as a guide on how to design cell culture studies, perform cell culture methods, or a document about good cell culture practices. With each review we fear that the manuscript is moving away from its intended original structure and aim. Our assessments were not about the credibility of the claims, we just want to point out whether a method, if done properly, can be utilized to measure something or not. Obviously, there are many factors affecting the overall reliability of your findings including: the number of samples, how the components of the assays as well as the controls were prepared, where and how did the samples and assay components were stored, was the cell line authenticated, was the right carrier control, right blind, right positive control, right negative control used, what was the room temperature, did antibiotics affect the findings etc. We are not trying to evaluate validity of the work we investigated, and we are not trying to assess their quality. That is why we focused on the claims and methods. However, there were a few exceptions where we had to investigate the experimental design itself. All those were listed as “conditional” in the table 1. If we were to look into the details experimental design and question the experimental conditions for each, then every entry would be “conditional”, and main points we are trying to make becomes lost. However when we re-write parts of the discussion section according to the recommendations of the reviewers we included a condensed general good practices recommendations (paragraph: 293-306).

Reviewer #4: The article revised by Ozkaya and Geyik is improved. In particular I appreciate the addition of discussion around the caveats of the study and assays discussed. However, I find that the article has gained length rather than clarity, and that the authors are still missing an opportunity to provide clear recommendations for good cell culture technique (as it pertains to all cell-based assays) and optimal assay use to address specific biological questions accurately. While I see value in drawing attention to widespread incorrect or over-interpretation from basic assays, the potential for impact lies in being explicit about what would have been needed for a claim to be deemed sufficient. This information is only included in the Methods and is incomplete (examples are provided), the authors could consider adding a summary table of assays and the conclusions that can be drawn from them. Although I will leave it to the editor to decide whether or not this falls within the scope of the current paper.

We just want to point out again that the aim of this work is not to provide readers with recommendations for good cell culture technique. Although we agree that the article has gained length rather than clarity, it is mostly because the reviewers wanted to turn this work into something we never intended it to be. Consequently, with each revision, we are just trying to change the manuscript based on the recommendations without altering the core. And the core is just about drawing attention to widespread incorrect or over-interpretation of some basic assays. We are very grateful for the invaluable recommendations, and it is really a pleasure to see that there are researchers out there as enthusiastic as we are about in-vitro practices. However, we are neither ready nor willing to turn this work into a guide on good cell culture practices.

We believe that the most important part of a scientific paper is the methods section. Placing the information regarding our approach on that section was a deliberate decision. We believe that methods part must contain all the information required to understand what was done in a study. Therefore, we placed both definitions we embraced along with our strategy to evaluate the claims in that section. The reviewer considers this information as incomplete because it does not include information regarding how to conduct and correctly interpret each method. However, we do not want to include those in this study. However, we did include some recommendations (paragraph: 293-306)

Reviewer #4: I continue to disagree with the authors’ assessment that ‘viability measurement is the only purpose of an MTT assay that is somewhat acceptable’, when really it is evaluating changes in metabolic activity in a population of cells. The same number of cells can have vastly different MTT readouts based on treatment, and while I agree that using these assays to assert cell death is particularly problematic, the oversimplification is detrimental to the paper. An increase in signal does not always indicate an increase in cell number.

We have made our points on this in the previous revision. We would like to leave the final decision on this to the editor.

Reviewer #4: Remove references to the discussion from the Methods section.

References were removed.

Reviewer #4: I do not understand why cell count and DNA content are only considered sufficient evidence for proliferation in the absence of treatment. Background levels of cell death need to be assayed under all conditions, controls included.

We agree with the necessity of evaluating the basal level of death. However, studies we investigated did not measure background cell death when they claim proliferation. That is why we considered only the absence of treatment where the effects of cell death on the measurement is minimal, as acceptable.

Reviewer #4 Why is ‘Real-time cell imaging’ (which I assume to refer to time-lapse microscopy) insufficient for quantifying cell death? Watching cells over time is both a straightforward and accurate way to monitor cell fate.

There was just one study where the “real-time cell imaging” was used to evaluate cell death (#49). In that study researchers just counted the number of viable cells over time. The graph was presented as “%viability” but they refer the data as an indicator of cell death.

Reviewer #4 DAPI is membrane permeable, how is it correctly used to test membrane integrity? It can be used for DNA content, and typically is used to complement a membrane impermeable dye to accurately evaluate the fraction of cell death.

DAPI can be membrane-permeable depending on the concentration and the incubation period but its rate of absorption by the cells is greatly increased in non-viable cells and fixated cells compared to viable ones. Therefore, it can be used to measure membrane integrity.

1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4496231/

2. https://pubmed.ncbi.nlm.nih.gov/27037070/

Reviewer #4 Same concern for acridine orange and Hoechst in the Viability section of the table. Unless they are used with GhostDye (or PI etc), or DNA content is accounted for, they can only yield overall cell counts.

In all the cases, acridine orange and Hoechst were used along with an indicator of cell death:

• Hoechst with PI (#60)

• Acridine orange with EtBr (#68)

• Acridine orange with PI (#77)

• Acridine orange with PI (#77)

• Acridine orange with DAPI (#77)

Reviewer #4 Please clarify the difference between ‘Dye inclusion’ and ‘Membrane integrity’.

Dye inclusion: methods utilizing dyes which are taken up by and indicate viable cells. Viable cell dyes.

Membrane integrity: methods utilizing dyes which can indicate the loss of membrane integrity. Non-viable cell dyes.

Reviewer #4 Propidium iodine should be Propidium iodide.

Corrected

Reviewer #4 pH3 positive cells represent a small fraction of a population and can represent cells arrested in M-phase, their presence alone does not reflect the overall proliferative potential of a population.

In the study (#95) researchers accepted “lack of pH3 positive cells” as an indicator of halted proliferation. The logic is: “if there are no cells in M-phase in a group of cells, it means that the population is not proliferating.”. That is why we considered their evidence as sufficient. We included this explanation in Table 1.

Reviewer #4 Change in proliferation is not limited to decrease in rate, perturbations can also stimulate proliferation.

We were not trying to claim that the change to be limited to the decrease. We have made changes in the manuscript to clarify this.

Reviewer #4 The discussion needs to be re-worked and copy edited. In particular lines 297-310. The discussion of IC50 values does not seem to fit – although these methods aim to improve reproducibility of large-scale drug response studies, this is somewhat separate from the issues the current paper is trying to address.

That paragraph almost entirely consists of sentences added as a response to the previous criticism provided by the reviewers. We agree that it is disconnected. We have re-written the part as suggested. IC50 part was removed.

Attachment

Submitted filename: Ozkaya 3rd response to reviewers.docx

Click here for additional data file. (21.2KB, docx)

Decision Letter 3

Hamidreza Montazeri Aliabadi

3 Jan 2022

From viability to cell death: claims with insufficient evidence in high-impact cell culture studies

PONE-D-21-12647R3

Dear Dr. Ozkaya,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Hamidreza Montazeri Aliabadi

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #3: We appreciate the extensive revisions that have been completed by the author. While we continue to disagree on some of the finer points of this topic, these are discussions that can play out in the peer-reviewed literature. They have done a commendable job in ensuring all their data is available for any further discussions that other authors would like to have. I think this will be an interesting discussion point for further considerations.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No

Acceptance letter

Hamidreza Montazeri Aliabadi

17 Jan 2022

PONE-D-21-12647R3

From viability to cell death: claims with insufficient evidence in high-impact cell culture studies

Dear Dr. Özkaya:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Hamidreza Montazeri Aliabadi

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Data

    (CSV)

    Click here for additional data file. (77.6KB, csv)
    Attachment

    Submitted filename: renamed_ab46f.docx

    Click here for additional data file. (14.3KB, docx)
    Attachment

    Submitted filename: Ozkaya Response to Reviewers.docx

    Click here for additional data file. (27KB, docx)
    Attachment

    Submitted filename: PLOS ONE R1 review.docx

    Click here for additional data file. (15.8KB, docx)
    Attachment

    Submitted filename: Ozkaya 2nd Response to Reviewers.docx

    Click here for additional data file. (25KB, docx)
    Attachment

    Submitted filename: Ozkaya 3rd response to reviewers.docx

    Click here for additional data file. (21.2KB, docx)

    Data Availability Statement

    All relevant data are within the manuscript and its Supporting Information files.

    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES