An integrated methodological framework for the validation and verification of clinical testing by qRT-PCR

Abstract

This paper outlines a practical method for validating quantitative-qualitative techniques used to detect genetic material through qRT-PCR, specifically focusing on SARS-CoV-2 testing and adhering to ISO/IEC 17025:2018 accreditation standards. Despite the prevalence of quantitative-qualitative screening in genetic testing, comprehensive validation guidelines remain a notable gap in the field. Such guidelines could be applied to other molecular testing areas that rely on these techniques, particularly those involving sample handling, automated extraction, and amplification processes, which can significantly impact results. This work describes the statistical approaches behind qRT-PCR protocols, followed by a technical characterization profile of the validation process. Modifications to the gold standard method allowed us to establish a technical limit of detection (LOD) of 5,09 copies/reaction at a 95 % confidence interval.

Highlights

• •This manuscript expands the knowledge about the validation and verification methods for clinical testing
• •Proposes a pragmatic approach for in-house validation of quantitative and qualitative methods for genetic material detection.
• •Provides relevant considerations for optimizing internal procedures to ensure good bias control in clinical testing

Glossary

Amplification efficiency

the rate at which the target DNA sequence is multiplied during a PCR reaction, as determined from a standard curve [1]

Accuracy

How close a test result is to accepted value [2]

Quantification Cycle (Cq)

the point in a PCR reaction where the fluorescent signal becomes detectable above background noise [3]

Cutoff

The smallest amount of a substance that can be measured [4]

R²Coefficient

the statistical relationship between cycle threshold and log-transformed DNA concentration [1]

Dynamic Range

the range of concentrations over which a test can accurately and precisely measure a substance [1]

Repeatability

the consistency of test results when the same test is performed multiple times on the same sample within the same laboratory [5]

Reproducibility

the consistency of test results when the same test is performed on the same sample in different laboratories [5]

Sensitivity

the ability of a test to correctly identify positive cases, especially those with low concentrations of the target analyte [6]

Specificity

The test's ability to distinguish between true negatives and false positives, ensuring accurate negative results [6]

Validation

the documented process of demonstrating that a method is fit for its intended use by evaluating its performance characteristics [3]

Verification

the process of confirming that a previously validated method performs as expected in a different laboratory setting [3]

Precision

the degree of agreement between independent measurements of the same quantity obtained under the same conditions [1]

Trueness

the comparison of measurement results obtained from a laboratory's method to those obtained from certified reference materials [1,3]

Negative Predictive Value (NPV)

the probability that a person truly does not have a disease, given a negative test result [7]

Positive Predictive Value (PPV)

the likelihood of actual disease presence following a positive diagnostic test outcome [7]

Uncertainty

a statistical parameter expressing the range of values within which the true value of a measurement lies [5]

1. Introduction

New pathogen emergence is particularly likely in the context of socioeconomic inequality, habitat fragmentation, and urbanization. Pathogens have the potential to spread and cause pandemic occurrences like the recent SARS-CoV-2 outbreak [8]. Emerging agents typically have limited diagnostic tools, and therefore, the process for epidemiologic intervention through surveillance systems and the validation of diagnostic methods workflows is a continuous process to be implemented [9]. When the WHO declared COVID-19 a global pandemic on March 11, 2020, the urgent necessity for reliable laboratory diagnostic tests for SARS-CoV-2 became evident [10,11].

Two widely used qRT-PCR procedures for detecting SARS-CoV-2 are the protocols developed by the Charité-Universitätsmedizin Berlin Institute of Virology and the Centers for Disease Control and Prevention (CDC) respectively [12,13]. The Charité protocol targets the sequences of E and RdRP genes [10]. Generally, the E gene is evaluated as a confirmatory tool because it detects all viruses from the Sarbeco virus subgenus. The RdRP gene is commonly used as confirmatory testing (specific to SARS-CoV-2) [10]. Otherwise, the CDC protocol tracks three N genes (N1, N2, and N3) on the SARS-CoV-2 sequence [11]. The Charité protocol was standardized and validated using the qRT-PCR method, and due to the nature of the methodology and its level of reliability in the detection of the virus, this was transferred to the National Reference Laboratories of Latin America through the Pan American Health Organization (PAHO) [14]. Subsequently, different diagnostic methods have been assembled in several countries, some of which have been approved by the Food and Drug Administration (FDA) and marketed in our country under the INVIMA sanitary registration.

This globally recognized protocol has a workflow that reliably detects and further discriminates 2019-nCoV from SARS-CoV, falling within a LOD of 5.2 copies of RNA/reaction, with a confidence interval (CI) of 95 %, ranging from 3.7 to 9.6 RNA copies/reaction [10]. However, standardized protocols should be adapted to the available technological infrastructure and laboratory capabilities through a comprehensive workflow for verification and validation. Our approach seeks to perform qualitative and quantitative analyses, where validation criteria include sensitivity, accuracy, veracity, reproducibility, and robustness among others. The validation of this protocol in our facilities is crucial, especially when using high-performance automated solutions, as false positive or negative results can have serious consequences for patient care [13].

During a pandemic, automated molecular diagnostics solutions can manage a large volume of samples and swiftly rule out or validate suspected cases [13]. On this matter, we survey an integrated approach to conduct in-house and inter-laboratory validation using a high-throughput automated format to ensure the method is fit for diagnostic purposes during internal handling under ISO 17025:2018 requirements (Fig. 1). Additionally, the analytical performance of the Charité-Universitätsmedizin Berlin Institute of Virology protocol was evaluated on an automated platform, providing detailed step-by-step analysis, of the samples that tested positive (Fig. 2). On this matter, EU Regulation No. 625/2017 stipulated that laboratories must validate standard methods before implementing new tests or calibrations. This validation process ensures the accuracy and reliability of the methods. Furthermore, in the event of any modification or adaptation to a standard procedure, the laboratory must revalidate the method to maintain its quality standards. This requirement aligns with the guidelines outlined in section 5.4.2 of the ISO/IEC 17025:2018 standard [15].

Fig. 1.

Standardized framework for validating and verifying SARS-CoV-2 molecular testing to assess the method's performance and fitness.

Fig. 2.

Validation/verification general workflow addressed by each operator, considering the processes that may contribute to measurement uncertainties. The image was created with BioRender.

2. Materials and methods

Ethics approval and consent to participate. The upper respiratory tract samples used for this study were obtained through the Universidad Industrial de Santander (Santander, Colombia). All samples were obtained by health authorities using informed patient consent and processed by a laboratory fixed to the medicine program. All procedures were conducted in accordance with relevant ethical guidelines and regulations, including the Declaration of Helsinki. The study was approved by the University of Applied and Environmental Sciences-U.D.C.A, Ethics Committee for Institutional Research (Session No. 52).

Safe sample handling and processing. The reception, handling, protection, storage, conservation, and disposal of the samples were carried out following the internal operative processes according to the ISO/IEC17025:2018 standards. In addition, all equipment used in the process considered able to affect the results was calibrated, qualified, and submitted to maintenance protocol following the ISO/IEC 17025:2018 standards [16]. The viral inactivation was performed as follows the manufacturer's instructions (Thermo Scientific) by adding 5 μL of Proteinase K to 200 μL of analytical samples previously approved to proceed. 10 μL of extraction control was also pre-mixed (70 pb fragment of EAV from Equine Arteritis Virus was used as Internal PCR Control) with the sample. The DNA/RNA of 200 μL of negative control from prostate cells (5 × 10⁴ PC3 cells), positive Q controls (Positive samples containing 10,000 digital copies (dC)/dL, provided as a liquid-frozen tube, Ref: SCV2QC01-A, Qnostic), and respiratory tract samples were extracted under controlled conditions. Importantly, the target concentration of the Q-panel has been designed to fall in the middle of the dynamic range. Samples were processed at different timelines with MagMAX™ Viral/Pathogen II Nucleic Acid Isolation Kit (Thermo Fisher Scientific), according to the manufacturer's instructions, and using the automated nucleic acid extractor Kingfisher Flex System (Thermo Fischer Scientific). All plates and reagents described in Table 1 were processed inside the biosafety cabinet.

Table 1.

Processing plates (200-μL specimen volume) preparation (adapted from the MagMAX™ Viral/Pathogen II Nucleic Acid Isolation guide). The processing plates were prepared as follows.

Plate ID	Plate position	Plate type	Reagent	Volume per well
Wash Plate 1	2	96 Deep-Well Plate	Wash Solution	500 μL
Wash Plate 2	3		80 % Ethanol solution	500 μL
Elution Plate	4		Elution Buffer	50 μL
Tip Comb Plate	5	Tip Comb

After plate preparation, samples were mixed up with binding beads mix (265 μL of binding buffer + 10 μL of microbeads), subsequently, the sample was vortexed to ensure a homogeneous bead mixture. Negative controls were always included in all experimental procedures (Negative viral cell control (NVCC); non-infected human cells + EAV control, extraction reagents (ERC) + EAV control; Water (WC) + EAV control). The extracted genetic material concentration and purity were calculated by measuring the absorption at 260/280 nm with a UV spectrophotometer (Biodrop, Biorad).

The LightMix® Modular SARS kit (TIB Molbiol, Berlin, Germany) and LightCycler Multiplex RNA Virus Master Mix (Roche, Basel, Switzerland) were used for qRT-PCR, following the manufacturer's instructions. A 20 μL reaction mixture was prepared, containing ultrapure water, Roche Master Mix, reagent mix, and RT enzyme. Ten microliters of the template were added to this mixture, and the qRT-PCR was performed on a LightCycler 96 Real-Time PCR System (Roche). Thermal cycling was carried out at 55 °C for 10 min for RT, followed by 95 °C for 3 min and then 45 cycles of 95 °C for 15 s, 58 °C for 30 s, including a cycling quantification mode in single. Primers and probes, as well as optimal concentrations, are listed in Table 2.

Table 2.

Gen E primers and probes (Corman et al., 2020).

Primer	Sequence	nM Concentration
E_Sarbeco_Forward	ACAGGTACGTTAATAGTTAATAGCGT	400
E_Sarbeco_P1, internal probe	FAM-ACACTAGCCATCCTTACTGCGCTTCG-BBQ	200
E_Sarbeco_Reverse	ATATTGCAGCAGTACGCACACA	400

3. Statistical for quantitative parameters

The extraction process for genetic material can inadvertently co-purify contaminants that can interfere with the PCR reaction, leading to reduced or absent amplification. Therefore, laboratories must ensure that their extraction procedures effectively eliminate these inhibitors to guarantee reliable PCR results [17].

Robustness based on ΔCq values. To assess the robustness of a qRT-PCR method, it's crucial to evaluate its performance under varying conditions. One approach involves diluting samples to identify potential inhibitors, as Cq values are directly linked to the initial target concentration. By slightly modifying experimental conditions, such as using a fractional factorial design, researchers can systematically study the impact of these changes on the results. This helps ensure the reliability and reproducibility of the qRT-PCR method [18]. Instead of examining modifications individually, our proposed approach introduces multiple changes simultaneously. To assess the impact of these changes, we analyzed two dilutions (12 % and 60 %) for each RNA extraction replicate. By calculating the difference in average Cq values between these dilutions (ΔCq = most diluted − most concentrated sample) and comparing them to the theoretically expected ΔCq values (Table 3), we were able to evaluate the effects of the modifications (Table 3). As the theoretical ΔCq value is 2 for 1:4 dilution, our acceptance criterion was: 1.5 < ΔCq < 0.5 [19]. Additionally, the variation coefficients were calculated to observe the degree of variability, where a higher CV means a greater dispersion (Table 3) [20].

Table 3.

Data collection and statistical analysis. The acceptance criteria were established by the variation coefficient (excellent values of CV < 0.05), the difference between ΔCq measured and the reported value (acceptable values less than <0,5).

Dilution %	SAMPLE	CqE1	CqE2	CqE3	($\bar{\mathit{X}}$)	SD	ΔCq measured	Acceptance criteria ≤ 0,5	CV.P	Absolute deviation (AD) (n–x) E1	AD (n–x) E2	AD (n-x)E3	AD Sum (E1 + E2 + E3)	∗U =(Sum/n x (n-1)
65	M1	28,8	29,9	28,8	29	0,53	2,5	0,00	0,02	−0,39	0,75	−0,35	0,00	0,00
12	M1	31,7	32,3	31	32	0,52			0,02	−0,02	0,65	−0,63	0,00	0,00
65	M2	18,3	19,7	19,4	20	0,11	2,32	−0,18	0,01	−1,29	0,11	−0,11	−1,29	−0,22
12	M2	20,4	22	21,7	22	0,16			0,01	−1,44	0,16	−0,16	−1,44	−0,24
65	M3	19,1	20,3	20	20	0,5	2,21	−0,29	0,03	−0,67	0,52	0,15	0,00	0,00
12	M3	21,5	22,5	22	22	0,41			0,02	−0,49	0,51	−0,02	0,00	0,00
65	M4	17,4	19,1	19,2	19	0,83	2,31	−0,19	0,04	−1,17	0,52	0,65	0,00	0,00
12	M4	20,4	21,3	21	21	0,35			0,02	−0,47	0,38	0,1	0,00	0,00
65	M5	12,4	14,3	13,2	13	0,79	2,52	0,02	0,06	−0,9	1,02	−0,12	0,00	0,00
12	M5	15,7	16,6	15	16	0,66			0,04	−0,1	0,85	−0,76	0,00	0,00
65	M6	21,9	23,9	22	23	0,92	2,51	0,01	0,04	−0,68	1,3	−0,63	0,00	0,00
12	M6	24,3	25,8	25,2	25	0,63			0,03	−0,81	0,73	0,08	0,00	0,00
65	M7	17,6	19,2	19,1	19	0,76	2,14	−0,36	0,04	−1,08	0,57	0,5	0,00	0,00
12	M7	20,2	21,3	20,8	21	0,46			0,02	−0,55	0,57	−0,01	0,00	0,00
65	M8	19,4	19,5	20,7	20	0,61	3,95	1,45	0,03	−0,46	−0,4	0,86	0,00	0,00
12	M8	26,7	22,1	22,7	24	2,03			0,09	2,86	−1,71	−1,14	0,00	0,00
65	M9	19	19,2	18,2	19	0,44	2,57	0,07	0,02	0,24	0,38	−0,61	0,00	0,00
12	M9	21,4	21,9	20,8	21	0,44			0,02	0,05	0,51	−0,56	0,00	0,00
65	M10	19,1	18,6	19,6	19	0,4	2,89	0,39	0,02	0,02	−0,49	0,48	0,00	0,00
12	M10	21,8	21,1	23,1	22	0,84			0,04	−0,23	−0,89	1,12	0,00	0,00
65	M11	27,4	29,3	0	29	0,92	2,08	−0,42	0,03	12,79	14,63	−14,63	12,79	2,13
12	M11	29,3	31,7	33	31	1,56			0,05	−2,08	0,4	1,67	0,00	0,00
65	M12	21,5	22,7	23,4	23	0,78	2,6	0,10	0,03	−1,02	0,15	0,86	0,00	0,00
12	M12	24,2	25,2	26	25	0,74			0,03	−0,93	0,04	0,89	0,00	0,00
65	M13	28		29,6	29	0,8	2,42	−0,08	0,03	−0,8	−28,84	0,8	−28,84	−4,81
12	M13	30,2	31,7	31,9	31	0,73			0,02	−1,03	0,44	0,6	0,00	0,00
65	M14	27,3	27,7	28,8	28	0,67	2,66	0,16	0,02	−0,66	−0,25	0,92	0,00	0,00
12	M14	29,5	30,7	31,5	31	0,83			0,03	−1,08	0,13	0,95	0,00	0,00
65	M15	26	27	25,4	26	0,65	2,68	0,18	0,02	−0,11	0,84	−0,74	0,00	0,00
12	M15	28,5	29,6	28,4	29	0,53			0,02	−0,37	0,74	−0,38	0,00	0,00
65	M16	14,3	15,3	14,9	15	0,43	1,98	−0,52	0,03	−0,55	0,5	0,06	0,00	0,00
12	M16	16,3	17	17	17	0,34			0,02	−0,48	0,25	0,23	0,00	0,00
65	M17	28	25,9	25,2	26	1,15	2,1	−0,40	0,04	1,58	−0,44	−1,14	0,00	0,00
12	M17	29,9	27,4	28,1	29	1,06				1,44	−1,07	−0,37	0,00	0,00

Limit of detection and quantification. An experimental design for qRT-PCR validation and verification study was applied to evaluate LOD and LOQ values. Determining LOD and LOQ is crucial to assessing the performance of any molecular diagnostic test, especially at low analyte concentrations. Rigorous statistical analysis is necessary to ensure that the analytical procedure itself does not introduce significant variability or bias into the results [21]. Additionally, LOD and LOQ can be determined based on the instrument's detection capabilities [22]. To address this, a subset of 1:10 serial DNA dilutions (twelve total replicates; 4 operators, and 3 replicates each) was prepared per operator until seven data points containing 96, 9.6, 9.6, 0.96, 0.096, 0.0096, and 0,00096 copies per reaction. The validation procedures were conducted for a quantitative PCR method designed to detect and quantify the SARS-CoV-2 E Gene in human respiratory fluids, following the recommendations of the Colombian National Institute of Health. All protocols were implemented in accordance with the requirements and procedures outlined in ISO/IEC 17025:2018 and the World Organization for Animal Health (OIE) guidelines [6]. The E gene qRT-PCR validation protocol is a three-step process that involves developing the qRT-PCR assay, characterizing its performance, and then evaluating the entire analytical process from sample extraction to PCR analysis [23]. The characterization of both the qRT-PCR assay and the entire analytical process involved determining acceptable values for slope, R², and %E. This was achieved by using a positive control (LightMix® SarbecoV E-gene plus EAV control, TIB MOLBIOL) under specific experimental conditions outlined in the design (Fig. 1, Fig. 2). To determine the lowest detectable and quantifiable amounts of the analyte, the limits of detection (LOD) and quantification (LOQ) were calculated using the formulas LOD = 3.3σ/S and LOQ = 10σ/S. The standard deviation of the response, σ, was estimated from the variability in the blank samples or the calibration curve data, specifically the standard error of the regression or the standard deviation of the y-intercept. The slope of the calibration curve, S, was obtained from the linear regression [21].

Bias and linearity uncertainty. Several factors can influence the analytical performance and measurement uncertainty, including cell type, sample matrix, sample treatment, cell lysis reagents, thermal conditions, standard curve generation, and DNA quality [24]. Therefore, when designing validation experiments, it is essential to consider the relevant factors that can contribute to variability in the results (Fig. 2). According to this, calculating bias (Supporting Information, Table 2) for each dilution level led us to find a critical bias value and the linearity uncertainties (ULINi) for each dilution (Supporting Information, Table 2), to evaluate the performance of linear regression for E gen positive control (Fig. 4). ULINi specify the linearity uncertainty determined for each dilution calculated from standard deviation (SD) and mean bias by the formulas Ulini = √〖SD〗² + 〖Cq average〗²; Ulin = ∑ 〖ULini〗²/# positive dilutions. The acceptable level of bias, which is determined by the laboratory, is typically around 0.25 log10 [21,23,24]. The difference between the lowest and highest values within a 0.5 log10 range (measured in log10 copy number) typically represents the acceptable bias. Finally, the ULIN value was used to compare the performance of qRT-PCR in our laboratory. Additionally, LOQ can be calculated by crossing this value such as the lowest concentration with a bias of 0.25 log10 used for the linearity range [23,25]. The collection of R programming codes for this section along with the outputs of interest such as acceptance requirements and linear uncertainties are provided in Supplementary Information 2. The data processing was performed with the R programming language (R Core Team 2023) through the RStudio Graphical User Interface (Posit Team 2023) (Supplementary Information 2).

Fig. 4.

Standard curves for qRT-PCR assays.

4. Qualitative analysis

The definition of performance parameters is crucial for qualitative analysis. Sensitivity, for instance, measures the assay's ability to correctly identify true positive samples. It is calculated as the ratio of true positive samples identified by the assay (a) to the total number of true positive samples (a+c), expressed as a percentage. This can be assessed both within a laboratory (intra-laboratory) and between different laboratories (inter-laboratory proficiency testing (Sensitivity = a/(a+c)) (Table 6) [4]. Cohen's Kappa Score was calculated using the formula k=(P0-Pe/1-Pe), where P0 represents the observed agreement calculated as the sum of true positives and true negatives divided by the total number of samples (Po = True Positives (TP) + True Negatives (TN)/N (samples number)). Pe, the expected agreement by chance, is calculated by multiplying the probability that both raters agree on a positive classification by the probability that both agree on a negative classification: Pe= ((Pe(rater 1 says Yes)/N) x ((Pe(rater 2 says Yes)/N)) + ((Pe(rater 1 says no)/N)) x ((Pe(rater 2 says no)/N)) [26]. Finally, observed and expected agreement were used to calculate Cohen's Kappa as follows; Kappa score = (Po – Pe)/(1 – Pe). The reproducibility was assayed by measuring the consistency of the results derived from assays performed by different operators (Intra-laboratory assay) and by interlaboratory comparison with the certified provider of Laboratory Quality Control in Brazil and Latin America through a proficiency test for SARS-CoV2 (Controllab). To ensure sample stability and homogeneity, samples used for multiple analysis rounds were divided into aliquots and stored appropriately. Importantly, a quality control (SARS-CoV-2 Q control 1, Qnostic) was used to monitor the molecular assay and to support relevant evidence under regulatory requirements of the standard ISO 17034. The target concentration of the control panel was designed to fall within the dynamic range of most molecular assays, ensuring consistent performance across different batches. Each lot of the control panel had a concentration of 10 copies/μL, with 10 μL aliquots containing 100 copies used for ten-fold serial dilutions.

Table 6.

Kappa statistic: a statistical measure used to assess the level of agreement between two or more raters when classifying categorical data, such as positive or negative results.

Confusion Matrix (Sensitivity)
Intra-lab	R1 \ R2	Positive	Negative	Total
	Positive	17,0	0,0	17,0
	Negative	0,0	3,0	3,0
	Total	17,0	3,0	20,0
Inter-lab	Lab2 \ Lab1	Positive	Negative	Total
	Positive	17,0	3,0	20,0
	Negative	0,0	3,0	3,0
	Total	17,0	6,0	23,0
Kappa Results section
	Intra-lab		Inter-lab
P0	1,000		0,870
Pr(Lab 1)	0,850		0,870
Pr(Lab 2)	0,150		0,130
Pe	0,745		0,677
∗Kappa	1,000		0,597
SE Kappa	0,000		0,199
(95 %) IC Kappa	1,0000	1,0000	0,2066	0,9867

∗ A kappa value of 1.00 represents perfect agreement between raters, meaning they classify all data points identically. A value of 0.00 suggests that the agreement between raters is no better than chance, indicating random classification. Conversely, a kappa value of −1.00 signifies complete disagreement, where raters consistently classify data points into opposite categories [26].

5. Results and discussion

Quantitative Analysis. 25 samples were processed once the optimal working conditions were verified. Each experiment was performed at least in triplicate at different timelines by different operators. The extractions were quantified and used for the amplification processes on different timelines under the same conditions, previously standardized in our laboratory with gen E positive control. Gene material quantifications were made for each extraction so that they reflect the deviations, relative deviations, and associated uncertainties for the extraction and the amplifications carried out. Extracted DNA/RNA samples were diluted to test the effect of inhibitors, which are often reduced at lower DNA/RNA concentrations (acceptance criteria ≤0,5, Table 3) [27]. In light of this, robustness is typically evaluated during validation by adjusting critical factors like reagent amounts and observing the impact [28]. For our qRT-PCR method, we examined the effects of varying sample dilutions. This test enables us to assess the method's ability to withstand errors caused by pipetting techniques and other systematic factors. Furthermore, the results of this ruggedness test can be used to define the method's operational boundaries, such as acceptable incubation time ranges during DNA/RNA extraction. In addition, by systematically varying key parameters or reagent concentrations, we can assess the method's sensitivity to perturbations. In the case of our qRT-PCR method, we focused on the impact of sample dilution on the overall performance. By understanding these limitations, we can optimize the method's performance and ensure reliable and accurate results [28]. Additionally, this method is considered a critical procedure in the validation/verification process in such a way that the ΔCqs were calculated for each amplification derived from 3 different extractions (Table 3). 16 samples were considered to have acceptable values (ΔCq <0,5; CV.P < 0,05). A theoretical ΔCq for a 1:5 dilution is 2.5. The majority of automatically extracted samples exhibited ΔCq values closely approximating this theoretical value. Ideally, the difference between the theoretical ΔCq (2.5) and the observed ΔCq should be less than 0.5. When DNA is diluted, the inhibitory effects of contaminants are often diminished or eradicated at lower DNA concentrations. This reduction in inhibition allows for a more accurate assessment of reaction efficiency by comparing the theoretical Cq of an uninhibited, undiluted sample to its experimentally measured Cq. This comparison offers valuable insights into the quality of the genetic material. However, in specific instances, inhibitory compounds may remain bound to DNA fragments even after dilution, leading to a lower number of amplifiable DNA/RNA copies than anticipated based on the nominal DNA/RNA concentration. This phenomenon is further illustrated in (Table 3). The technical procedure fits the robustness of the analytical method measuring its capacity to remain little affected by considerable dilutions, indicating its reliability during its routinary implementation (Fig. 3B). Furthermore, the samples that fit in the established acceptance criteria, normally continue the flow of analysis established in the laboratory (ΔCq <0,5; CV < 5 %). Notably, the presence of inhibitors was measured by qRT-PCR since It allows obtaining an ΔCq value from the concentrations worked greater than and/or equal to 0.05 (Table 3). In addition, the data sets also show a small relative uncertainty indicating a very accurate result (Table 3, U values). On the other hand, a regression curve was generated for each DNA dilution set by plotting all the Cq values obtained from each concentration level, for each operator, against the logarithmic transformation of the DNA copy number multiplied by 1000. This approach enabled the visual representation of the relationship between the Cq values and the corresponding DNA concentrations, facilitating the analysis of potential variations in measurement accuracy and precision across different operators and dilution levels (Fig. 4). By employing regression analysis, it was possible to quantify the slope and intercept of the regression line for each DNA dilution set, providing valuable information about the efficiency and sensitivity of the qPCR assay (Fig. 4). Additionally, Fig. 3A illustrates the influence of employing different extraction kits, which can slightly change de Cq values. This observation underscores the importance of standardizing extraction protocols to minimize inter-kit variability and ensure consistent and reliable results. Furthermore, to rigorously assess the reproducibility, repeatability, and combined uncertainty, along with the linearity bias associated with each operator, calibration curves were constructed using the RNA positive control for the E gene (Fig. 4). These calibration curves served as a critical tool for evaluating the performance of the qPCR assay. By subjecting these curves to a stringent set of acceptance criteria, including R2 values ranging from 0.95 to 1.0, amplification efficiencies between 95 % and 105 %, and slope values between 3.2 and 3.4, it was possible to identify and select the most suitable curves for subsequent statistical analysis. On this matter, the inclusion of these robust validation procedures is essential for ensuring the accuracy, precision, and reliability of clinical testing [29]. By carefully considering factors such as extraction kit variability and operator-specific performance, it is possible to mitigate potential sources of error and bias, ultimately leading to more accurate and reliable diagnostic results [29].

Fig. 3.

Robustness. We performed a robustness assay for our method measuring the capacity of Cq values to remain little affected by the introduction of different extraction kits (A) and serial sample dilutions (65 % and 12 % sample concentration) (B). Samples are denoted by their respective number.

LOD and LOQ 95 % estimation. We adopted different strategies to estimate the LOD and LOQ values (Table 4, Table 5). Still, a straightforward and pragmatic method to determine both LOD and LOQ for a qRT-PCR assay involves serial dilutions while assessing analytical sensitivity [21,30]. The 3.3σ/S method is widely used and is based on the statistical distribution of noise in the blank signal (Table 4). The 3.3 factor is derived from the assumption that a signal-to-noise ratio of 3 is required for reliable detection, assuming that the noise in the system is normally distributed and that the calibration curve is linear over the entire range of interest [31]. Dilution points were used to graph linear models based on the concentration-response curve (Fig. 4), the concentration at which 95 % of targets were positive was 0,50,915 copies/μL (equivalent to 5,0915 copies/reaction, Table 4). Subsequently, the concentration corresponding to the estimated LOQ at 95 % CI was 1,5 copies/reaction (0,149 copies/μL multiplied by 10 μL per reaction, Table 4). On the other hand, linear uncertainty assessment with the most dilute sample for which all three replicates per operator yield positive results allowed us to estimate similar results: a LOQ = 4058 Copies/μL with an intra-laboratory uncertainty associated with batches of ±0,981 (Table 5). This cut-off value, visually represented in Fig. 4, provides a reliable lower boundary for detecting the target analyte with confidence. It provides a more accurate estimate of the uncertainty in the concentration measurement, especially for low-concentration samples. By establishing the conservative LOD (LOD = 5,09 copies/reaction, Table 5) we ensure that subsequent analyses are sufficiently sensitive to identify even low levels of the target, minimizing the risk of false negative results [21,30]. In light of the results, while the number of replicates is the primary determinant of precision, concentration increments between the control samples are significant for LOD and LOQ accuracy [32]. On this matter, this method calculates the uncertainty in the concentration measurement based on the propagation of error from the uncertainty in the slope, intercept, and measurement error [23]. The LOQ value represents the lowest concentration of a substance that can be reliably measured and can only be determined for target concentrations or amounts that were explicitly included in the control sample. In addition, the LOQ is contingent upon the quality of the calibration curves used to establish the relationship between the concentration of a substance and the measured response [33]. These calibration curves must meet rigorous statistical criteria, including a high coefficient of determination (R²), acceptable slope values, and low percentage error (%E). A high R² value indicates a strong linear relationship between concentration and response, while acceptable slope values ensure that the calibration curve is not too steep or too shallow. Additionally, low percentage error (%E) signifies minimal measurement uncertainty [21,33].

Table 4.

Statistical analysis for the LOD and LOQ at a 95 % significant level.

	Operator 1	Operator 2	Operator 3	Operator 4
Regression Equation	Y = −3.038∗X + 41.09	Y = −3.380∗X + 41.09	Y = −3.171∗X + 42.02	Y = −3.155∗X + 42.35
SE Y-intercept	0,6948	0,5042	0,7091	0,3126
SD Y-intercept= (SE x √n)	1,2034	0,8733	1,2282	0,5414
LOD = (3,3 x SD/Slope)	−1,3072	0,8712	1,2846	0,5663
LOQ = (10 x SD y-intercept/slope)	3,9613	2,64	3,8929	1,7161
Intra Laboratory LOD and LOQ 95 %	Log (x)		Anti-log = Copies/μl
Average LOD 95 % (Copies/μl)	−1,007342115		0,509151258
Average LOQ 95 % (Copies/μl)	−3,052551864		0,149078947

Table 5.

Limit of Detection and Quantification. Based on the parameters such as Standard Error Y-intercept (SE Y-intercept) resolved in the linear regression per operator, the intra-laboratory LOD (95 %) value was calculated by applying the following equation LOD = 3.3x σ/S, where S is the slope of the calibration curve and σ is the standard deviation of the response [45].

Parameter	Value
N	12
Operators Ct average at 0,096 copies/reaction	35,98
Ct average ∗cutoff at 0,096 dilution = Ct average per operator +(2 x SD)	37,10
∗∗U Lin	0,98
∗∗∗LOD Copies/μL (Total copy number = 10(Ct-b)/m))	5,09
∗∗∗∗SD	0,36
BSL3-LAB ∗∗∗∗∗(LOQ)	4.058 ± 0,711
Significant levels section
Confidence Level	LCL	UCL
95,00 %	4.905	5.275
99,00 %	4.829	5.351

∗ The cutoff value is defined as a value that marks the lower limit of gen E detection; ∗∗ Lineal Uncertainty; ∗∗∗ Limit of detection, ∗∗∗∗ Standard Deviation; ∗∗∗∗∗ Limit of quantification. LCL; lower confidence limit, UCL; upper confidence limit.

On the other hand, interlaboratory tests, also known as proficiency testing or external quality assessment, are crucial for clinical testing laboratories for several reasons, including maintaining quality standards, ensuring accuracy and reliability, and promoting the standardization process [34]. On this matter, the discordances found in two samples in the inter-laboratory test (Table 6), were mainly regarding samples with low positivity signals (Cq > 35) but also, could be due to frequent sample degradation [35]. Additionally, validation revealed small differences in the Cq values for the analytes when using different diagnostic kits (Table 7), which confirmed that the Cq values slightly vary with different amplification strategies. Similar results were reported when calibration curves were carried out with a certified QC panel, specially designed to fall in the middle of the dynamic range (100-1 copies per reaction), which were consistent across batches and confirmed the loss of signal-detecting E gene close to 1 copies/per reaction in 10-fold dilutions, while a dilution containing 10 copies per reactions, was keeping signal in all experiments (Supporting information, Fig. 1).

Table 7.

Expanded uncertainty ΔU calculation.

Controllab test dates (Inter-laboratories)	Cq values for positive control gen E	n-average	(n-average)2	Sum	Variance relative to the average = Sum/n-1	SD	U type A	Relative RSDr
July 05–2021	28,13	0,015	0,000225	0,225	0,037	0,193	0,192	0,687
November 11–2021	28,04	0,105	0,011,025
November 11–2021	27,89	0,255	0,065,025
January 31–2022	28,4	0,255	0,065,025
February 02–2022	28,01	0,135	0,018,225
February 02–2022	28,4	0,255	0,065,025
Cq Average	28,145
aExpanded uncertainty ∪ = Δ∪ x 2	0,385
bΔm = [cm−CRM]	−0,255
cΔ∪ = √(U2m + u2CRM)	0,192
d∪ m=(SDr)x(√ (n))	0,26

^aThe expanded uncertainty ΔU, corresponding to an approximate 95 % confidence level, is calculated by the product between the standard uncertainty (Δu) and the coverage factor (k = 2).

^bΔm represents the difference between the average measured value and the reference value. Here, cm denotes the measured value, and CRM signifies the reference value.

^cCombined uncertainty between the result and certified value (= uncertainty of Δm).

^d∪m is the uncertainty of the measurement result, SDr is the standard deviation of the repeatability, and (n) is the number of independent measurement results [5].

The validation process yielded optimal results following the acceptance criteria suggested by the UNE/EN ISO/IEC 17025:2018. A critical aspect of the validation process was ensuring that the qPCR phase occurred within the dynamic range of the standard curve. This is essential for accurate and reliable quantification of target nucleic acids [36]. To enhance the precision and reproducibility of the assay, the analysis of calibration curves was replicated at least 12 times (Table 5). This rigorous approach minimized experimental variability and strengthened the statistical significance of the results. Furthermore, the study incorporated both intra- and inter-laboratory testing [37]. Intra-laboratory testing involves multiple operators within the same laboratory analyzing the same samples at different time points and assessing the consistency of results within a specific laboratory setting. Inter-laboratory testing, on the other hand, involves different laboratories analyzing the same samples, and evaluating the comparability of results across different laboratories [37]. By implementing these comprehensive validation procedures, the study aimed to establish the robustness and reliability of the qPCR assay, providing a solid foundation for its application in future research and clinical settings [36]. The performance of the method was also evaluated through efficiency percentage, which was found between 90 and 105 % in most cases, to check that the qRT-PCR amplification was working as expected. The method was found to be reliable and repeatable because the CV was consistently less than 10 % [28,38]. Furthermore, since sensitivity shows how well the process can identify the target, it was evaluated as part of the validation process. Moreover, the LOQ was calculated from the standard curves by computing the standard deviation of the replicate samples across different concentration levels [32]. The SD of the data was determined for each case using either a linear scale (relative quantities) or a log (Cq values), representing the average difference between the measured values (Table 5). To understand the spread of the data relative to its central tendency, we calculated the relative standard deviation (RSDr) or the coefficient of variation (CV = 100 × SD/mean) by scaling the standard deviation (SD) as a percentage of the mean.

Qualitative analysis. While specific sample size calculations are not routinely established for comparative analyses, a panel comprising 17 positive and three negative samples was employed to evaluate the correlation and concordance correlation coefficients, along with their respective confidence intervals. These statistical measures were used to quantify the agreement between the new test and a comparator test, providing valuable insights into the accuracy and reliability of the new or modified method. Additionally, the kappa value was calculated to further assess the level of agreement between the two tests, offering a comprehensive evaluation of their concordance [39]. The kappa statistic is a versatile metric employed for categorical data, specifically designed to quantify the degree of agreement between two raters or methods, beyond what would be expected by chance. By accounting for the possibility of random agreement, the kappa statistic provides a more accurate assessment of the true level of concordance between the two variables. A higher kappa value indicates a stronger level of agreement, suggesting that the two methods or raters are highly consistent in their assessments (Table 6) [39]. The latest was used to assess the degree to which, two or more raters examined the same data in our lab (Table 6). Perfect agreement is indicated by a kappa value of 1.00; no agreement beyond what would be expected by chance is indicated by a value of 0.00; and complete disagreement is indicated by a kappa value of −1.00 [39]. The kappa value is derived from the data organized in a 2 × 2 contingency table, which is a tabular format used to record the outcomes of the comparative test (Table 6). The results comparing operators and laboratories showed satisfactory kappa values ranging between perfect and moderate agreement (1000 and 0,596 respectively) (Table 6). The accuracy of the data collected for this study in representing the true values of the variables measured, both within and between laboratories, is crucial. This accuracy, referred to as rater reliability, is therefore highly significant in this context. To ensure the reliability of our laboratory's coronavirus identification testing, we implemented a rigorous quality control measure. This involved conducting three rounds of in-house verification through an inter-laboratory proficiency test program. This program assesses the laboratory's ability to consistently produce accurate and reliable results, thereby enhancing the overall quality and validity of the study's findings. (Table 7). As part of our statistical performance for each critical process, we performed a reasonable, fit-for-purpose estimation of uncertainty (U), as shown in Table 5, Table 7 While uncertainty is often associated with individual measurements, it is important to recognize that in the context of qRT-PCR, the overall uncertainty can be influenced by various factors, including sample handling and analysis procedures. To address this, our laboratory conducted a comprehensive evaluation of the specific uncertainty associated with our in-house validation process under well-defined conditions. By focusing on the potential sources of variability within our laboratory's workflow, we aimed to minimize the impact of these factors on the final results. The tables referenced (Table 5, Table 7) provide a detailed breakdown of the uncertainty components and their contributions to the overall uncertainty [39]. Fig. 5 visually illustrates the potential sources of uncertainty, highlighting the importance of careful sample handling and analysis to mitigate their impact. By implementing robust quality control measures and adhering to standardized protocols, our laboratory strives to minimize uncertainty and ensure the highest level of accuracy and reliability in our qRT-PCR testing. It is well known that all components of uncertainty, including systematic effects contribute to the dispersion [40]. Importantly, after the measurement of a CRM (PC, Cat. 40-0776-96, TIB MOLBIOL) we calculated the difference between the mean experimental value and the certified value based on interlaboratory results as follows Δm = [cm−CRM], If so, Δm is the absolute difference between mean measured value and certified value, and the uncertainty of Δm is Δu, which provide the combined uncertainty of result and certified value. Finally, the expanded uncertainty (Δu) at 95 % CI was calculated by the product between Δu and the coverage factor k = 2 [40]. Our results showed that the absolute difference Δm was ≤ UΔ, suggesting that there is no significant discrepancy between the measured value and the certified value, confirming that our method does not exhibit significant bias (Table 7) [40].

Fig. 5.

Confidence and prediction bands offer visual cues to quantify uncertainty. A) 95 % confidence bands delineate the region within which we are 95 % confident that the true underlying curve resides (depicted in green). 95 % prediction bands encompass the area where we anticipate 95 % of future observations will fall, accounting for both the inherent variability in the data and the uncertainty associated with the estimated curve. B) The residual plot illustrates the discrepancy between each data point and the corresponding value predicted by the curve. Positive residuals indicate data points that lie above the curve, while negative residuals represent points below the curve. Data visualization was executed using GraphPad Prism 8 software.

Verification. Verification studies typically confirm that an assay maintains its expected performance when used consistently within the laboratory. To verify analytical sensitivity (LOD), 10-fold dilutions of a single reference or positive control were tested. No additional testing for analytical sensitivity or specificity is recommended. According to this, practical settings for the verification of a qRT-PCR method included dynamic range in terms of R², amplification efficiency, robustness, RSDr, LOD, and LOQ calculation [40,41].

According to the ISO definition, in this work, we evaluated the precision, robustness, linearity, accuracy, and analytical sensitivity of the RT-qPCR-based assay [42]. On this matter, we used this assay to rapidly detect SARS-CoV-2 in a BSL-3 Laboratory under controlled conditions. We first analyzed the robustness with potential issues that may occur due to the presence of inhibitors based on the effect of dilutions affecting Cq values, due to an improper pipetting of operators. To evaluate how these problems would impact the diagnosis, 20 SARS-CoV-2 derived samples were processed at different times, by different operators, and, in some cases, samples were analyzed with two different kits (interlaboratory test) (Fig. 3). In addition, acceptance/rejection criteria were critical for the record of our results which showed very accurate results with ΔCq values below 0,5 and a coefficient of variation less or equal to 5 % (Table 3). The observed dynamic range for the gen E from the SARS-CoV-2 presented logs with an approximated LOD of 5,09 copies/reaction, with PCR efficiencies between 95 % and 105 %, and the replicates of the normalized curves were very similar between batches (Fig. 4, Fig. 5). The acceptable ranges of error may be obtained from the relevant proficiency tests or the current validation studies in our process of verifying and validating the accuracy of a modified FDA-approved test (which alters the extraction protocol by an automated procedure) (Table 7, Fig. 4, Fig. 5) [43]. Importantly, as we lack established criteria in some steps of the process, we expressed the imprecision as the target value plus SDs and their respective uncertainty U values. However, current recommendations state that the accuracy surrounding the mean value should not be greater than 15 % of the CV, except at the LOQ, where precision should not exceed 25 % (RSDr values) [42,44].

Considering quantitative analysis, the genetic material assays provided a technical report outlining the estimation of measurement uncertainty (MU). To this subject, laboratories should give supporting evidence of how MU was estimated in the quantitative results obtained by qRT-PCR [40]. In this regard, the selected workflow addressing the evaluation of different parameters leads us to conclude that our method performed under BSL-3 conditions is suitable for detecting SarsCov2.

Conclusion. qRT-PCR is still one of the primary diagnostic techniques used today. However, it is crucial to consider how a test is used in particular circumstances, and it is important to evaluate the procedures' limitations and performance levels [24]. This study provides the results of internal validation and comparison of qRT-PCR assays for SARS-CoV-2 testing under a selected workflow, to evaluate different parameters which leads us to conclude that our method carried out in a BSL-3 context, is suitable for detecting SarsCov2. The implemented method with small modifications allowed us to establish a technical LOD of 5,09 copies/reaction at a 95 % confidence interval, with a detection range between 4905 and 5,27 copies/reaction. In addition, the intra-laboratory uncertainty associated between batches was established at ± U (0,981) (Table 5). On this matter, our findings show high similarity to the gold standard method developed by Corman et al., where a technical LOD was about 5.2 copies/reaction at a confidence interval of 95 %, with a detection range between 3.7 and 9.6 copies/reaction [10]. In light of the current investigation, it appears reasonable to define cutoff values based on internal validation, regardless of the various laboratory conditions that turn out to be comparable with the procedures currently approved by the FDA and WHO.

CRediT authorship contribution statement

Carolina Cardona-Ramírez: Writing – review & editing, Writing – original draft, Validation, Supervision, Methodology, Formal analysis, Data curation, Conceptualization. Cruz Elena Enríquez-Valencia: Validation, Data curation, Conceptualization. Gina Méndez-Callejas: Writing – review & editing, Validation, Methodology, Formal analysis, Data curation. Giovanna Meza Barreto: Validation, Formal analysis, Conceptualization. Gabriel Andrés Tafur-Gómez: Formal analysis, Conceptualization. Danny Wilson Sanjuanelo-Corredor: Visualization, Formal analysis, Conceptualization.

Data availability statement

All data generated or analyzed during this study are included in this published article and its supplementary information files.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: