Comment to the article by Pedro Brotons: Validation and implementation of a direct RT-qPCR method for rapid screening of SARS-CoV-2 infection by using non-invasive saliva samples, IJID 110 (2021) 363–370

Dear Editor,

We read with interest the article entitled “Validation and implementation of a direct RT-qPCR method for rapid screening of SARS-CoV-2 infection by using non-invasive saliva samples” (Brotons et al., 2021). This study validates and implements an optimized screening method for the detection of SARS-CoV-2 ribonucleic acid, integrating the use of self-collected raw saliva samples, single-step heat-treated virus inactivation and ribonucleic acid extraction, and direct reverse transcription-quantitative polymerase chain reaction (RT-qPCR). Although this article provides valuable information, we believe that when the authors evaluated the diagnostic accuracy of saliva-based direct reverse transcription-polymerase chain reaction (RT-PCR) versus standard RT-PCR on nasopharyngeal swabs, some results are worth discussing. We noticed that the agreement of the two assays was not assessed.

Generally, the question of agreement, or consistency among samples collecting data immediately arises due to the variability in different diagnosis methods. Thus, well-designed research studies must consequently incorporate procedures that measure agreement among the various data collectors (McHugh, 2012). There are a number of statistics that have been used to measure inter-rater reliability. A partial list includes overall accuracy (Asai et al., 2022), Cohen's kappa (McHugh, 2012), Pearson's R (Salvagno et al., 2021), Spearman Rho (Geisler et al., 2020), intraclass correlation coefficient (Rezaeipandari et al., 2022), concordance correlation coefficient (Campana et al., 2022), Krippen-dorff's alpha (Dupuis et al., 2021), and Matthews correlation coefficient (Qorri et al., 2022). Here, we will only consider the most common measures, Cohen's kappa and overall accuracy (Table 1 ).

Table 1.

Weighted kappa value and overall accuracy for calculating agreement between saliva-based direct RT-qPCR and standard RT-PCR on nasopharyngeal swab

Standard RT-qPCR	Saliva-based direct RT-qPCR					Overall accuracy
k = 0.802 (strong agreement)		Positive	Negative	Inconclusive	Total	97.36%, (22+273+0)/303
	Positive	22	0	1	23
	Negative	0	273	0	273
	Inconclusive	1	6	0	7
	Total	23	279	1	303

Note: The data has been cited from the article published by Brotons et al. (2021) and undergone modification. k is the weighted kappa value calculated by us.

RT-qPCR, reverse transcription-quantitative polymerase chain reaction.

Generally, Cohen's kappa statistic is suitable for evaluating two raters (McHugh, 2012). In Cohen's kappa statistic, weighted kappa statistic should be used to calculate the inter-rater reliability in the presence of more than two categories (Li et al., 2022).

Weighted kappa is calculated as follows:

$$k_w=1-\frac{{\sum }_{i=1}^n{\sum }_{j=1}^n{w}_{ij}{p}_{ij}}{{\sum }_{i=1}^n{\sum }_{j=1}^n{w}_{ij}{p}_i{q}_j}$$ (1)

The value of $u_{jj}\left(i{i}^{\prime }\right)$ is the proportion of objects put in the same category j by both raters $i$ and $i^{\prime }$. The value of $p_{ij}$ is the proportion of objects that rater $i$ assigned to category $j$. According to McHugh (2012), the kappa result should be interpreted as follows: 0-0.20 indicating no agreement, 0.21-0.39 as minimal agreement, 0.40-0.59 as weak agreement, 0.60-0.79 as moderate agreement, 0.80-0.90 as strong agreement, and 0.91-1.00 as almost perfect agreement.

Therefore, according to the authors’ data, the weighted kappa value between saliva-based direct RT-qPCR and standard RT-PCR on nasopharyngeal swab evaluated by us was 0.802 (95% confidence interval = 0.669-0.935), indicating a strong agreement. The overall accuracy between the two assays was 97.36%.

CRediT authorship contribution statement

Tianfei Yu: Writing – original draft. Fangfang Liu: Data curation. Haichang Yin: Data curation. Nana Yi: Data curation. Ming Li: Writing – review & editing.