Skip to main content
NCBI home page
PLOS One logoLink to PLOS One
. 2022 Jul 21;17(7):e0271808. doi: 10.1371/journal.pone.0271808

The importance of effect sizes when comparing cycle threshold values of SARS-CoV-2 variants

Celine Brinkmann 1,2, Peter Gohl 3, Dietrich Mack 3, Johannes Pfeifer 4, Mònica Palmada Fenés 2, Oliver Harzer 3,5, Bernhard Zöllner 1,*
Editor: Etsuro Ito6
PMCID: PMC9302753  PMID: 35862414

Abstract

Purpose

We aimed to elaborate whether cycle threshold (Ct) values differ significantly between wild type SARS-CoV-2 (wtV) and certain viral variants and how strong or weak a potential significant effect might be.

Methods

In a retrospective study, we investigated 1873 SARS-CoV-2 positive samples for the occurrence of viral marker mutations. Age, gender, clinical setting, days after onset of symptoms, and Ct values were recorded. Statistical analysis was carried out with special consideration of effect sizes.

Results

During the study period wtV was detected in 1013 samples (54%), while 845 (45%) patients carried the Alpha variant of concern (VOC), and 15 (1%) the Beta VOC. For further analysis, only wtV and the Alpha VOC were included. In a multi-factor ANOVA and post-hoc test with Bonferroni-correction for the age groups we found significant main-effects for Ct values of the viral variant (wtV mean 26.4 (SD 4.27); Alpha VOC mean 25.0 (SD 3.84); F (1,1850) = 55.841; p < .001) and the clinical setting (outpatients: mean 25.7 (SD 4.1); inpatients: mean 27.0 (SD 4.2); F (1,1850) = 8.520, p = .004). However, since the effect sizes were very small (eta squared for the Alpha VOC = .029 and the clinical setting = .004), there was only a slight trend towards higher viral loads of the Alpha VOC compared to wtV.

Conclusions

In order to compare different variants of SARS-CoV-2 the calculation of effect sizes seems to be necessary. A combination of p-values as estimates of the existance of an effect and effect sizes as estimates of the magnitude of a potential effect may allow a better insight into transmission mechanisms of SARS-CoV-2.

Introduction

The new pandemic coronavirus SARS-CoV-2 has the capacity to select for mutational variants. Particularly variants of concern (VOC) pose a considerable threat because they exhibit increased transmissibility, detrimental change in COVID-19 epidemiology, an increased virulence or change in clinical disease presentation or a decreased effectiveness of public health and social measures or available diagnostics, vaccines and therapeutics [1, 2].

Currently, five VOCs have been described in the European Union: the Alpha, Beta, Gamma, Delta, and Omicron variant of concern [3]. In January 2021, almost exclusively wild-type SARS-CoV-2 (wtV) circulated in Germany. However, in the further course, it was more and more replaced by the Alpha VOC, reaching a 88% prevalence of this variant at the end of March 2021 [4]. During this period, it was possible to compare the properties and kinetics of wild type and mutated SARS-CoV-2 in a direct way.

Cycle threshold (Ct) values that are generated in a real-time polymerase chain reaction (PCR), are a way of estimating the virus concentration in a sample semi-quantitatively. This roughly reflects the viral load in the patient’s material and could allow conclusions to be drawn about the infectiousness of a patient [5].

The Ct values of the individual VOCs were examined and compared in some studies [6–9]. The results of these investigations were that significant differences between Ct values of VOCs could be determined. However, statistical significance is only one part of the equation. While statistical significance shows that an effect exists in a study and is unlikely to be caused by chance, effect sizes describe whether the effect is large enough to be meaningful in the real world [10]. Particularly with large sample sizes it is more likely that significant results with low p-values can be found in quantitative studies [11]. Therefore, reporting both p-values and effect sizes is much more informative and relevant for the situation in vivo than reporting only p-values. For this purpose, effect sizes can be calculated within different statistical models. In ANOVA, eta squared (η2) is well suited for the calculation of the effect size and gives an estimate how strong the variation, e.g. of the viral load, can be accounted to each factor or predictor. This makes it possible to classify which factor is more or less important.

Our study aimed to investigate whether there was a significant difference in Ct values between wild-type SARS-CoV-2 (wtV) and its mutational variants of concern. Particularly the influence of multiple cofactors like age, gender, days from onset of symptoms, and clinical setting on Ct values were analysed with special consideration of the magnitude of the effect.

Material and methods

Subjects and samples

The study period of this retrospective analysis lasted from January 20th until March 17th 2021. All positive samples in the even-numbered weeks, regardless of the measured Ct-values, and a collective of samples with a Ct value <30 in the uneven numbered weeks were included. Dry swabs without transport medium were taken from the patients´ naso- or oropharyngeal area, shipped to our laboratory within 24–48 hours at room temperature and processed within a mean time of 22 hours. For all patients, age, gender, time of sampling, in- or outpatient status, and Ct values were documented. In telephone calls with the institutions that sent positive samples, the time between patient´s onset of symptoms and sampling was determined.

SARS-CoV-2 PCR and determination of SARS-CoV-2 VOCs

PCR was run on a cobas® 8800 instrument (Roche Diagnostics, Mannheim, Germany) using the cobas® SARS-CoV-2 test (also Roche Diagnostics, Mannheim, Germany), which includes specific detection of the ORF1 a/b non-structural region and E-gene. Pool tests were not performed, each sample was tested individually. To ensure (i) the stability of the viral RNA during transport and (ii) the reproducibility of our PCR system we investigated three previously tested samples with different Ct values. These samples were stored at room temperature and retested on three consecutive days. Thereafter, the day-to-day coefficients of variation were calculated for each of these samples.

SARS-CoV-2 mutational variants were analysed with the cobas® SARS-CoV-2 Variant Set-1, which enabels the detection of the key mutations for the Alpha-, Beta-, and Gamma-VOCs (N501Y, E484K, and the 69–70 deletion).

Statistical methods

The Mann-Whitney U-test was applied in a first attempt using Abacus 2.0 (Labanalytics, 2021, Jena, Germany) to analyse Ct values in females and males as well as in- and outpatients and different age groups, as described previously [e.g. references 5 and 6]. Using this test, the Ct values ​​of wtV and the Alpha VOC for the respective groups were compared. Thus, two groups were formed for these demographic factors, each consisting of unpaired samples. Similarly, rates of hospitalized patients with wtV and the Alpha VOC were compared with Fisher´s exact test.

To avoid alpha-error-accumulation which occurs with e.g. multiple testing we ran a multi-factor Analysis of Variance (ANOVA)(IBM SPSS for Windows, Version 27.0, 2020, Armonk, NY, USA) afterwards and compared the results of both analyses. The ANOVA model term contained exclusively the main effects, no interactions were included. The factor “days after onset of symptoms” was excluded due to a low sample size. All the assumptions for the calculation, including equality of variances (F = 1.026; p = .427) were met. Post-hoc-test with Bonferroni-correction was calculated for the age factor, as it had more than two groups. A p-level of <0.05 was considered to be statistically significant.

The effect sizes were calculated with the designated standard function of SPSS (see in S1 Table). Values of η2 between 0.01 and 0.059 are considered as small effects, those between 0.06 and 0.139 correspond to a medium effect, and of 0.14 or greater to a large effect.

Results

A total of 1873 SARS-CoV-2 positive samples were analysed with Ct values ranging between 13–36. The reproducibility and stability of Ct values over a period of three days at room temperature was high: repeated measurements of three samples on three consecutive days showed a coefficient of variation of 3% (sample 1, Ct value 17), of 2.2% (sample 2, Ct value 23), and of 2.3% (sample 3, Ct value 28).

Wild type virus was detected in 1013 samples (54%), 845 (45%) patients carried the Alpha VOC, and 15 (1%) the Beta VOC. The distribution of patients between outpatients and inpatients was 954/59 for wtV, 817/28 for the Alpha VOC, and 15/0 for the Beta VOC, respectively. The prevalence of the Alpha VOC changed from 9% (41/464) in calendar week 4 to 74% (406/552) in calendar week 10 (Fig 1). The Beta VOC played a subordinate role with 0.9% (4/438) in week 8 and 2% in week 10 (11/552) (Fig 1).

Fig 1. Prevalence of all positives of wild type SARS-CoV-2 (wtV) and the Alpha and Beta VOCs.

Fig 1

Open in a new tab

Due to the low case number (n = 15), the Beta VOC was excluded from further analysis. Table 1 shows the results of the remaining samples (n = 1858) using a Mann-Whitney-U two-group comparison as described in other publications [e.g. references 5 and 6]. Significantly, lower Ct values of the Alpha VOC were detected for the factor gender (p<0.0001), outpatients (p<0.0001), and age groups 0–80 years (p<0.004). The rate of inpatients carrying the Alpha VOC (28/845) was significantly lower than those carrying wtV (59/1013, p = 0.011). The time between the onset of symptoms and the day of sampling could be determined cross-sectionally for a subgroup of 242 patients until day 5 of the infection. The results revealed only a weak significance on day one showing slightly lower Ct values of the Alpha VOC compared to wtV (25.0 (SD 3.47) vs. 26.8 (SD 3.54), p = 0.03) (Table 1). Days 2–5 showed no significant differences in Ct values between wtV and the Alpha VOC. Hence, this parameter was excluded from the subsequent multiple factor analysis.

Table 1. Ct values and standard deviations (SD) of wild type SARS-CoV-2 (wtV) and the Alpha variant of concern (VOC) with p-values derived from the associated Mann-Whitney-U tests.

Factor (cases: wtV/Alpha VOC) Mean Ct value (SD) p-value
wtV Alpha VOC
All (1013/845) 26.4 (4.27) 25.0 (3.84) <0.0001
Gender Female (534/406) 26.4 (4.32) 25.2 (3.86) <0.0001
Male (479/439) 26.5 (4.21) 24.9 (3.82) <0.0001
Setting outpatient (954/817) 26.4 (4.25) 24.9 (3.83) <0.0001
inpatient (59/28) 26.9 (4.57) 27.2 (3.47) 0.73
Age [years] 0–20 (116/123) 26.9 (4.44) 25.3 (3.93) 0.003
21–40 (296/290) 26.6 (4.19) 25.2 (3.67) <0.0001
41–60 (361/291) 26.2 (4.19) 24.5 (3.82) <0.0001
61–80 (176/120) 26.4 (4.32) 24.9 (4.04) 0.004
>80 (64/21) 25.3 (4.46) 25.6 (4.33) 0.8
Days from onset of symptoms (subgroup) 1 (26/63) 26.8 (3.54) 25.0 (3.47) 0.03
2 (19/51) 24.4 (3.32) 24.4 (3.72) 0.96
3 (19/22) 23.7 (3.11) 25.0 (4.11) 0.24
4 (9/19) 26.4 (3.60) 24.4 (3.32) 0.16
5 (5/9) 24.2 (4.15) 24.2 (3.15) 0.99
Open in a new tab

To avoid the multiple comparison problem of testing, and further investigate the relevance of all factors together, we employed a multi-factor ANOVA. This model showed significant main effects on Ct values for the factors viral variant (F (1,1850) = 55.841, p < .001), age (F (4,1850) = 3.297, p = .011), and setting (F (1,1850) = 8.520, p = .004). The gender groups did not differ significantly regarding the viral load. The effect sizes showed a small effect for the Alpha VOC (η2 = .029) as well as for the setting (η2 = .004) and age (η2 = .007), meaning that e.g. 2.9% of the variation in the Ct values account for the factor viral variant. Wildtype virus had slightly higher Ct values (mean 26.4, SD 4.27) compared to the Alpha VOC (mean 25.0, SD 3.84) and total inpatients had slightly higher Ct values (mean 27.0, SD 4.2) than total outpatients (mean 25.7, SD 4.12) (Fig 2). Post-hoc tests with Bonferroni-correction showed no significant differences between the age categories, so the small, significant main effect was neglected. Taken together, significant differences of Ct values could be found between wtV and the Alpha VOC and between each total in- and outpatients, however with only very small effect sizes.

Fig 2. Box-Whisker-Plots of Ct values for SARS-CoV-2.

Fig 2

Open in a new tab

(Left) Type of SARS-CoV-2. (Right) Clinical setting. Eta squared (η2) indicates the effect size.

Discussion

The investigation of viral variants of SARS-CoV-2 is becoming increasingly important. There has been a great effort to characterize and compare wild type SARS-CoV-2 and its mutational variants in vitro [12–17]. In these studies, increased infectivity and faster viral entry into target cells have been shown for VOCs compared to wtV. However, studies on the comparison in vivo are difficult to perform as wtV and its variant should circulate in the population at the same time. That is why there are only a few analyses with this approach [6–9, 18–22].

In our analysis, patients carrying the Alpha VOC had a mean Ct value that was one PCR cycle lower than patients carrying wtV, which is in line with previously published studies [6, 7]. Also, in the study by Luo et al. [8] the mean difference in Ct values between the B.1.2, Delta and Alpha variants was in the range of 1–2 PCR cycles (20.61 vs. 19.62 vs. 21.77, respectively). Although these and our results were statistically highly significant, the mean difference of 1–2 PCR cycles is low. P-values, however, are confounded by large sample sizes [11]. This could have been true for at least our study and was the reason why we calculated effect sizes. Also, there is another problem, namely the lack of standardization of the pre-analytical phase prior to the PCR for SARS-CoV-2. For instance, the technique of swabbing or the influence of transport conditions on individual samples could cause a bias in the evaluation of Ct values. One possible solution to balance out preanalytical errors in single samples is to analyse large patient groups, as it was done in our study. However, as mentioned above, this requires the specification of effect sizes in addition to the analysis of p-values.

We found lower Ct values of the Alpha VOC ​​in both sexes, in outpatients and almost all age groups. However, a subsequent multiple-factor analysis led to the result that there was only a significant difference between wtV and the Alpha VOC in the total cohort and between in- and outpatients concerning Ct values and, of great importance, that the effect size was low for these factors. Particularly, the low influence of the viral variant on Ct values was surprising, considering the observation of higher transmissibility of the Alpha VOC compared to wtV [6–9, 18–22]. A plausible explanation could be an increased binding affinity of this variant to the ACE2 receptor [14–16]. This leads to higher infectivity even with comparable virus concentrations [17].

Our study covered a period during which the Alpha VOC had almost completely replaced the wtV. The prevalence rates in our cohort correlate well to the general German prevalence rates determined by the Robert Koch Institute [4]. In addition, hardly any people were vaccinated at that time in Germany. Therefore, a direct comparison of Ct values in currently taken samples was possible and the results were not biased by different immune protection rates of infected persons.

Because the time since symptom onset is a key determinant of viral load in SARS-CoV-2 infections [23, 24], we tried to control for this cofactor. A limitation was, that we could only retrospectively record the days between onset of symptoms and sampling in a cross-sectional manner and that no longitudinal examination of the Ct values could be carried out in individuals during the acute phase of the infection. Nevertheless, our results implicate that apart from slightly lower Ct values of the Alpha VOC at day one there were no significant differences between wtV and VOC in the Ct values even during the first five days of the disease. This strengthens the hypothesis that the increased infectiousness of the Alpha VOC in vivo relies on other factors rather than high viral concentrations during the initial phase of the infection process. In vivo, factors like receptor-affinity [14–16] or enhanced target cell entry [12, 15] e.g. associated with altered spike cleavage sites [12] of the Alpha VOC seem to play a more important role.

Conclusions

For future studies elaborating Ct values of e.g. the Delta, Omicron (in particular BA.2 or BA.5) or other emerging VOCs, statistical analyzes with multiple factors seem to be essential. Additionally, we propose the calculation of effect sizes, which would allow an estimation of the magnitude of differences in Ct values ​​and of their true meaning in vivo. This could provide clues to the really decisive factors in such a complicated, multifactorial process as the infectivity and transmission of SARS-CoV-2.

Supporting information

S1 Table. Output of effect sizes in SPSS for the data of our study.

(PNG)

Click here for additional data file. (80.4KB, png)
S1 Dataset. Anonymized original dataset.

(PDF)

Click here for additional data file. (1.3MB, pdf)

Data Availability

All relevant data are within the paper and its Supporting Information files.

Funding Statement

The author(s) received no specific funding for this work.

References

  • 1.World Health Organisation. Tracking SARS-CoV-2 variants [Online]. 2020. Available: https://www.who.int/en/activities/tracking-SARS-CoV-2-variants/ (accessed on 7 September 2021). [Google Scholar]
  • 2.European Centre for Disease Prevention and Control (ECDC). 2021. SARS-CoV-2 variants of concern as of 8 July 2021 [Online]. Available: https://www.ecdc.europa.eu/en/covid-19/variants-concern (accessed on 7 September 2021). [Google Scholar]
  • 3.European Center for Disease Prevention and Control (ECDC). SARS-CoV-2 variants of concern as of 13 January 2022. https://www.ecdc.europa.eu/en/covid-19/variants-concern (accessed on 19 January 2022). [Google Scholar]
  • 4.Robert-Koch-Institut. Aktualisierter Bericht zu Virusvarianten von SARS-CoV-2 in Deutschland, insbesondere zur Variant of Concern (VOC) B.1.1.7. https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/DESH/Bericht_VOC_2021-04-14.pdf?__blob=publicationFile, [Accessed 03.September 2021]. [Google Scholar]
  • 5.Bullard J, Dust K, Funk D, Strong JE,Alexander D, Garnett L et al. Predicting infectious Severe Acute Respiratory Syndrome Coronavirus 2 from diagnostic samples. Clin Infect Dis. 2020;71(10):2663–2666. doi: 10.1093/cid/ciaa638 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Calistri P, Amato L, Puglia I, Cito F, Di Giuseppe A, Danzetta ML, et al. Infection sustained by lineage B.1.1.7 of SARS-CoV-2 is characterised by longer persistence and higher viral RNA loads in nasopharyngeal swabs, Int J Infect Dis. 2021;105:753–755. doi: 10.1016/j.ijid.2021.03.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Roquebert B, Haim-Boukobza S, Trombert-Paolantoni S, Lecorche E, Verdurme L, Foulonge V, et al. SARS-CoV-2 variants of concern are associated with lower RT-PCR amplification cycles between January and March 2021 in France. Int J Infect Dis. 2021;113:12–14. doi: 10.1016/j.ijid.2021.09.076 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Luo CH, Morris CP, Sachithanandham J, Amadi A, Gaston DC, Li M, et al. Infection with the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Delta variant is associated with higher recovery of infectious virus compared to the Alpha variant in both unvaccinated and vaccinated individuals. Clin Infect Dis. 2021;ciab986. doi: 10.1093/cid/ciab986 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Kidd M, Richter A, Best A, Cumley N, Mirza J, Percival B, et al. S-variant SARS-CoV-2 lineage B1.1.7 is associated with significantly higher viral loads in samples tested by ThermoFisher TaqPath RT-qPCR. J Infect Dis. 2021;223(10):1666–1670. doi: 10.1093/infdis/jiab082 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Bhandari P. What is effect size and why does it matter? Scribbr. 2022. [published 2020 Dec 22, revised 2022 Jun 01, cited 2022 Jun 08]. Available from: https://www.scribbr.com/statistics/effect-size/ [Google Scholar]
  • 11.Bakker A, Cai J, English L, Kaiser G, Mesa V, van Dooren W. Beyond small, medium, or large: points of consideration when interpreting effect sizes. Educ Stud Math. 2019;102:1–8. 10.1007/s10649-019-09908-4 [DOI] [Google Scholar]
  • 12.Meng B, Kemp SA, Papa G, Datir R, Ferreira IATM, Marelli S, et al. Recurrent emergence of SARS-CoV-2 spike deletion H69/V70 and its role in the Alpha variant B.1.1.7, Cell Rep. 2021;35(13):109292. doi: 10.1016/j.celrep.2021.109292 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Zhang J, Xiao T, Cai Y, Lavine CL, Peng H, Zhu H, et al. Membrane fusion and immune evasion by the spike protein of SARS-CoV-2 Delta variant. Science. 2021;374:1353–1360. doi: 10.1126/science.abl9463 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Kim S, Liu Y, Lei Z, Dicker J, Cao Y, Zhang XF, et al. Differential interactions between human ACE2 and spike RBD of SARS-CoV-2 variants of concern. J Chem Theory Comput. 2021;17:7972–7979. doi: 10.1021/acs.jctc.1c00965 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Kim Y, Gaudreault NN, Meekins DA, Perera KD, Bold D, Trujillo JD, et al. Effects of spike mutations in SARS-CoV-2 variants of concern on human or animal ACE2-mediated virus entry and neutralization. Microbiol Spectr. 2022; ahead of print. doi: 10.1128/spectrum.01789-21 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Starr TN, Greaney AJ, Hilton SK, Ellis D, Crawford KHD, Dingens AS, et al. Deep mutational scanning of SARS-CoV-2 receptor binding domain reveals constraints on folding and ACE2 binding. Cell. 2020;182:1295–1310. doi: 10.1016/j.cell.2020.08.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Hu J, He CL, Gao QZ, Zhang GJ, Cao XX, Long QX, et al. The D614G mutation of SARS-CoV-2 spike protein enhances viral infectivity and decreases neutralization sensitivity to individual convalescent sera. BioRrixv preprint 10.1101/2020.06.20.161323 [DOI] [Google Scholar]
  • 18.Tanaka H, Hirayama A, Nagai H, Shirai C, Takahashi Y, Shinomiya H, et al. Increased transmissibility of the SARS-CoV-2 Alpha variant in a Japanese population. Int J Environ Res Public Health. 2021;18:7752. doi: 10.3390/ijerph18157752 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Graham MS, Sudre CH, May A, Antonelli M, Murray B, Varsavsky T, et al. Changes in symptomatology, reinfection, and transmissibility associated with the SARS-CoV-2 variant B.1.1.7: An ecological study. Lancet Public Health. 2021;6:e335–e345. doi: 10.1016/S2468-2667(21)00055-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Davies NG, Abbott S, Barnard RC, Jarvis CI, Kucharski AJ, Munday JD, et al. Estimated transmissibility and impact of SARS-CoV-2 lineage B.1.1.7 in England. Science. 2021;372:6538. doi: 10.1126/science.abg3055 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Volz E, Mishra S, Chand M, Barrett JC, Johnson R, Geidelberg L, et al. Assessing transmissibility of SARS-CoV-2 lineage B.1.1.7 in England. Nature. 2021;593:266–269. doi: 10.1038/s41586-021-03470-x [DOI] [PubMed] [Google Scholar]
  • 22.Leung K, Shum MH, Leung GM, Lam TT, Wu JT. Early transmissibility assessment of the N501Y mutant strains of SARS-CoV-2 in the United Kingdom, October to November 2020. Eurosurveillance. 2021;26:2002106. doi: 10.2807/1560-7917.ES.2020.26.1.2002106 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Biguenet A, Bouiller K, Marty-Quinternet S, Brunel AS, Chirouze C, Lepiller Q. SARS-CoV-2 respiratory viral loads and association with clinical and biological features. J Med Virol. 2021;93:1761–1765. doi: 10.1002/jmv.26489 [DOI] [PubMed] [Google Scholar]
  • 24.Salvatore PP, Dawson P, Wadhwa A, Rabold EM, Buono S, Dietrich EA, et al. Epidemiological correlates of PCR cycle threshold values in the detection of SARS-CoV-2. Clin Infect Dis. 2021;72:e761–e767. doi: 10.1093/cid/ciaa1469 [DOI] [PMC free article] [PubMed] [Google Scholar]

Decision Letter 0

Etsuro Ito

7 Jun 2022

PONE-D-22-07729The importance of effect sizes when comparing cycle threshold values ​​of SARS-CoV-2 variantsPLOS ONE

Dear Dr. Zöllner,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Unfortunately, one of the two independent reviewers did not provide his comments.

Please submit your revised manuscript by Jul 22 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Etsuro Ito

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The study compares the Ct, a direct indicator of the viral load, of the wild type SARS-CoV-2 and and its mutational variants. In particular, it insists on the need to carry out the analysis of effect size in order to rigorously assess the possible differences.

The effect size which is the central statistic and the fundamental element of this publication is not explained enough. The introduction should give a sufficient ruling on this question and above all define what the size effect means and its important contribution to the resolution of scientific questions.

The statistics used should be better defined and explained. It is also important to indicate the software used, the versions and the years to carry out the statistical tests. For this section, also specify the level of significance for all the statistical tests carried out.

1) Line 101 How long were samples stored and in what media. Please precise the storage conditions because it can affect the stability of the virus genetic material and consequently the ct-values.

2) Line 109 Please were samples tested in pools or singly? Why did authors not include oropharyngeal samples?

3) Line 104. Can't transporting samples at room temperature alter the quality of the genetic material and make the results uncertain in favor of the samples that would have survived the temperature better?

4) Line 110, indicate manufacturer and country for cobas® 8800 instrument

5) Line 112: Write ORF1ab and E gene in italics

6) Line 129-129, it is not only the age factor that has more than 2 groups, the setting factor also

7) Line 123. Please clarify if the comparison of patient percentages between Wild, Alpha and Beta types SARS-CoV-2 is for hospitalized patients only? Does this include only in-patients or also out-patients?

8) Line 134. Why still talk about Ct range between 15-36 when it had been declared that only samples with Ct value <30 are included in the study (line 102). Please clarify

9) Line 139. The addition of all the cases (1013 wild type, 845 alpha VOC, 15 beta VOC) gives 1901 positive samples and not 1873 as declared in line 134. Please clarify

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2022 Jul 21;17(7):e0271808. doi: 10.1371/journal.pone.0271808.r002

Author response to Decision Letter 0

6 Jul 2022

Dear Prof. Ito,

thank you for giving us the opportunity to resubmit a revised version of our manuscript (manuscript no. PONE-D-22-07729) entitled

“The importance of effect sizes when comparing cycle threshold values of SARS-CoV-2 variants” by Celine Brinkmann, Peter Gohl, Dietrich Mack, Johannes Pfeifer, Mònica Palmada Fenés, Oliver Harzer and Bernhard Zöllner*

We would also like to thank the reviewer for the valuable comments. We very much appreciate the suggestions for improvement of our manuscript and revised the article according to the comments:

Journal requirements:

Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: No

We uploaded all anonymized data underlying the study when submitting the first version of the article (see file “Article_Brinkmann et al._data set.pdf”). Did something get lost?

Major point A:

„The effect size which is the central statistic and the fundamental element of this publication is not explained enough. The introduction should give a sufficient ruling on this question and above all define what the size effect means and its important contribution to the resolution of scientific questions.“

We now have added a paragraph into the introduction in which the problems of p-values and the advantage of reporting effect sizes are described (lines 88-101). In addition, the type of calculation and interpretation of the effect size is now described in the Material and Methods section (lines 152-155 and supporting information Table S1).

Major point B:

„The statistics used should be better defined and explained. It is also important to indicate the software used, the versions and the years to carry out the statistical tests. For this section, also specify the level of significance for all the statistical tests carried out.“

The statistics were defined and described in more detail in the section Material and Methods including software versions and years of release (lines 135-145). Also, the level of significance has been specified (lines 150-151) and the cut off values for the interpretation of eta squared have been inserted (lines 153-155). We have also included the printout of the ANOVA with the respective effect sizes as a supporting information (S1 Table, see also point 9) at the end of the manuscript. This makes our approach clearer for the reader.

Point 1) „How long were samples stored and in what media. Please precise the storage conditions because it can affect the stability of the virus genetic material and consequently the ct-values.“

This is an important point. Swabs without transport medium were processed immediately after arrival in our lab within a mean time of 22 hours. Given a transport time of 24-48 hours we evaluated the stability of the viral RNA over a 3 days period at room temperature and found a high stability of the results. This is mentioned in the section Material and Methods (lines 113-115 and 125-130) and Results (line 160).

Point 2) “Please were samples tested in pools or singly? Why did authors not include oropharyngeal samples?“

All samples were tested individually, no pool-testing was performed. This is now mentioned in line 125. In Germany, oro- or nasopharyngeal sampling is recommended and we actually received both types of swabs. We subsumed everything under nasopharyngeal in the first version of our manuscript but now clarified the point in the revised manuscript (line 114).

Point 3) “Can't transporting samples at room temperature alter the quality of the genetic material and make the results uncertain in favour of the samples that would have survived the temperature better?“

This question could actually represent a bias and is a problem for all studies evaluating Ct values. In our opinion, one way to at least partially compensate for such preanalytical errors in single samples is to analyse large patient groups. Since, however, a large sample size confounds the significance levels, effect sizes should also be calculated as a consequence. Hence, if you consider pre-analytical errors and want to compensate for this with larger patient cohorts, this is another argument for effect sizes. We took up the reviewer's concerns and included a brief paragraph on this possible bias in the Discussion section (lines 218-230).

Point 4) “Indicate manufacturer and country for cobas® 8800 instrument“

Both details have been added (lines 122-123).

Point 5) “Write ORF1ab and E gene in italics“

This has been implemented (lines 124 and 125).

Point 6) “It is not only the age factor that has more than 2 groups, the setting factor also.“

The clinical setting was divided in out- and inpatients, which are two groups.

Point 7) “Please clarify if the comparison of patient percentages between Wild, Alpha and Beta types SARS-CoV-2 is for hospitalized patients only? Does this include only in-patients or also out-patients?“

The distribution of patients by wild type virus, Alpha and Beta VOC applies to the entire cohort. We added a sentence to the result section with the exact distribution of in- and outpatients for these three virus types (lines 164-166), so that this point should now be clarified.

Point 8) “Why still talk about Ct range between 15-36 when it had been declared that only samples with Ct value <30 are included in the study (line 102). Please clarify“

We further clarified our misleading formulation in the text (lines 111-112).

Point 9) “The addition of all the cases (1013 wild type, 845 alpha VOC, 15 beta VOC) gives 1901 positive samples and not 1873 as declared in line 134. Please clarify“

We recalculated all data, but did not detect any mistakes. Adding 1013+845+15 gives 1873, but maybe we expressed something misleadingly. If so, we would be grateful for a hint. In addition, we discussed the results again in the authorship and came to the conclusion that the specification of eta squared is better suited than partial eta squared. Both work well as effect sizes in an ANOVA, but eta squared is considered easier to interpret. In order to show that both values differ only slightly and have no influence on the results or conclusions, we have included the table of the SPSS output as a supporting information (S1 Table) at the end of the manuscript.

In addition to the points mentioned above, we revised the manuscript according to the author's instructions and updated the reference list. We look forward to the review of our revised manuscript and hope that it is now considered acceptable for publication in PLoS One.

With kind regards,

Bernhard Zöllner

Attachment

Submitted filename: Article_Brinkmann et al._PLOS ONE_response to reviewers.docx

Click here for additional data file. (166.5KB, docx)

Decision Letter 1

Etsuro Ito

8 Jul 2022

The importance of effect sizes when comparing cycle threshold values ​​of SARS-CoV-2 variants

PONE-D-22-07729R1

Dear Dr. Zöllner,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Etsuro Ito

Academic Editor

PLOS ONE

Acceptance letter

Etsuro Ito

12 Jul 2022

PONE-D-22-07729R1

The importance of effect sizes when comparing cycle threshold values ​​of SARS-CoV-2 variants

Dear Dr. Zöllner:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Prof. Etsuro Ito

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Table. Output of effect sizes in SPSS for the data of our study.

    (PNG)

    Click here for additional data file. (80.4KB, png)
    S1 Dataset. Anonymized original dataset.

    (PDF)

    Click here for additional data file. (1.3MB, pdf)
    Attachment

    Submitted filename: Article_Brinkmann et al._PLOS ONE_response to reviewers.docx

    Click here for additional data file. (166.5KB, docx)

    Data Availability Statement

    All relevant data are within the paper and its Supporting Information files.

    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES