Analytical performance evaluation of a new integrated clinical chemistry and immunoassay analyzer

Abstract

Background

Clinical laboratories perform a wide range of tests that are used by healthcare professionals to guide medical decision making. Use of automated analyzers in the clinical laboratory can improve patient care by not only reducing the turn-around-time (TAT) of results but also improving accuracy of the reported results by reducing human error. The aim of this study was to evaluate the performance characteristics of a new automated laboratory instrument, the Atellica® CI Analyzer, Model 1900, over a 3-month period in a European laboratory setting.

Methods

Analytical performance of 17 analytes (13 chemistry and four immunochemistry) was assessed by evaluating repeatability and within-laboratory precision using anonymized remnant serum samples. Method comparison studies were performed on the Atellica CI Analyzer and the Roche cobas® 6000.

Results

Excellent precision was observed with coefficients of variation (CVs) less than 2 % for repeatability and less than 3 % within-laboratory imprecision for most analytes. Comparison of select assays with the cobas 6000 system resulted in correlation coefficients ranging from 0.980 to 1.000.

Conclusion

This is the first reported evaluation of the Atellica CI Analyzer in a clinical laboratory setting. The strong analytical performance of the Atellica CI Analyzer demonstrates that this instrument is suitable for routine clinical use.

1. Introduction

Clinical laboratories perform a wide range of tests, the results of which are used to aid in the diagnosis, treatment, and follow-up of a variety of diseases and medical conditions [1]. Given the key role that laboratory testing plays in guiding clinical decision making, laboratories are expected to provide clinically relevant, accurate, and timely results [2]. This demand must be met despite increasing pressures from shortages in both clinical laboratory personnel and financial resources. Thus, improvements in laboratory testing that increase efficiency and reduce cost while preserving the quality and accuracy of results are needed [3].

The new Siemens Healthineers Atellica® CI Analyzer, Model 1900 (Siemens Healthcare Diagnostics Inc., Tarrytown, USA) addresses many of these needs, offering the same planned menu and comparable technology to that of the larger Atellica® Solutions analyzer in a smaller footprint. Not only do the Atellica CI Analyzer and Atellica Solution have similar workflows and user interfaces, but they also utilize the same consumables, thereby increasing operational efficiency of the laboratory by allowing sharing of resources (both consumables and laboratory professionals) across laboratories within the same network.

Although analyzers are rigorously tested by the manufacturer to satisfy both local and global regulatory requirements, laboratories must still verify the performance of instruments prior to reporting patient results. The aim of this study was to evaluate the performance characteristics of the Atellica CI Analyzer, Model 1900 in a clinical laboratory setting. The performance evaluation included an assessment of precision and method comparison against the Roche cobas® 6000 for a subset of commonly tested analytes.

2. Materials and methods

2.1. Study design

The analytical performance evaluation and workflow studies on the Atellica CI Analyzer were performed at the Laboratory Dr. Limbach & colleagues in Heidelberg, Germany that belongs to the Limbach group SE (Heidelberg, Germany), which represents the largest network of laboratories in Germany and one of the largest clinical laboratory networks in Europe. Testing took place from February to May 2023. Only anonymized remnant patient specimens were used during these studies.

The Atellica CI Analyzer assay detection capabilities mirror the Atellica Solution system and include integrated multisensory technology (IMT), photometric, and turbidimetric technology for chemistry, as well as chemiluminescence with acridinium ester technology for immunochemistry. Analytical performance of the Atellica CI Analyzer was evaluated using 17 assays in total, representative of many of the most commonly ordered clinical laboratory tests: 13 for chemistry (Atellica CH Albumin BCG [ALB], Atellica CH Alkaline Phosphatase, Concentrated [ALP], Atellica CH Alanine Aminotransferase [ALT], Atellica CH Amylase _2 [AMY], Atellica CH Aspartate Aminotransferase [AST], Atellica CH Calcium [CA], Atellica CH Cholesterol_2 [CHOL], Atellica CH Glucose Hexokinase_3 [GLU], Atellica CH HDL Cholesterol [HDL], Atellica CH LDL Cholesterol [LDL], Atellica CH Triglycerides [TRIG], Atellica CH Uric Acid [UA], Atellica CH Urea Nitrogen [BUN], and four for immunochemistry (Atellica IM Protein-Specific Antigen [PSA], Atellica IM Total hCG [HCG], Atellica IM Free Thyroxine [FT4] and Atellica IM Thyroid Stimulating Hormone 3-Ultra [TSH]).

2.2. Precision

Precision on the Atellica CI Analyzer and Roche cobas 6000 (Roche Diagnostics, Germany) was performed in accordance with Clinical & Laboratory Standards Institute (CLSI) EP15-A3 guidelines using quality control (QC) material or patient pools. Briefly, three levels (L1, L2, and L3) of a lot-locked set of serum-based quality control (QC) material from Bio-Rad® InteliQ® Multiqual (ALB, ALP, ALT, AMY, AST, BUN, CA, CHOL, GLU, HDL, LDL, TRIG, UA) and InteliQ® Immunoassay Plus (HCG, PSA, FT4, and TSH) were run on the Atellica CI Analyzer and the Roche cobas 6000. Five replicates were measured per sample per day over 5 days (n = 25 measurements per sample).

Precision was also evaluated on the Atellica CI Analyzer using remnant anonymized serum patient samples obtained from routine laboratory testing performed by the Laboratory of Dr. Limbach & colleagues in Heidelberg, Germany. PSA and HCG were not included in this study. Fresh (never frozen) anonymized remnant serum patient samples with values in the normal range for each analyte were pooled from at least four unique individuals, aliquoted, and stored at 4 °C until use. This patient pool was assayed in singlicate once per day for five consecutive days.

Data were analyzed using a one factor (days) ANOVA model with Analyse-it software (Analyse-it Software, Ltd., Leeds, United Kingdom) for Microsoft Excel. Mean concentration, standard deviation (SD), repeatability, and within-laboratory precision were calculated for each analyte tested. Precision was expressed as coefficient of variation (CV%), calculated using the equation: CV% = (SD/mean of measured values) x 100 and are presented with bootstrapped-based 95 % confidence intervals (CIs).

2.3. Method comparison

Method comparison (MC) studies were performed according to CLSI EP09-A3 using remnant anonymized serum patient samples (n = ≥40) spanning each assay's analytical measuring range. Residual patient samples were run on the Atellica CI Analyzer and the cobas 6000 on the same day of collection. Data were analyzed using Passing-Bablok regression with 95 % CIs. Correlation coefficients of ≥0.95 were considered acceptable.

3. Results

3.1. Precision

Table 1, Table 2 summarize repeatability and within-laboratory CVs for chemistry (n = 13) and immunochemistry (n = 4) analytes, respectively, using QC material. For chemistry assays, Atellica CI Analyzer repeatability CVs ranged from 0 % to 1.9 % while within-laboratory CVs ranged from 0.5 % to 3.7 %. CVs for chemistry analytes on the cobas 6000 ranged from 0.5 % to 5.1 % for repeatability and 0.9 %–5.2 % for within-laboratory. Repeatability CVs for all Atellica CI Analyzer immunochemistry assays tested ranged from 1.1 % to 2.5 % and within-laboratory CVs ranged from 1.6 % to 3.8 %; repeatability CVs for immunochemistry analytes on the cobas 6000 ranged from 0.6 % to 2.3 % and within-laboratory CVs ranged from 1.6 % to 5.4 %. All results of the Atellica CI Analyzer precision studies were compared to precision data provided in the manufacturer's Instructions for Use (IFU) for each analyte and found to meet the manufacturer's specifications. Results for patient pooled precision demonstrated a within-laboratory CV of less than 3 % for all chemistry and immunochemistry analytes tested (Table 3).

Table 1.

QC repeatability and within-lab precision for chemistry analytes.

Red numbers indicate the highest observed values for repeatability and within lab %CVs.

Table 2.

QC repeatability and within-lab precision for immunochemistry analytes.

Red numbers indicate the highest observed values for repeatability and within lab %CVs.

Table 3.

Patient pool precision on the Atellica CI Analyzer.

Red number indicates the highest observed %CV.

Within-laboratory precision for each analyte was compared to quality specifications based on biological variation (BV) [4]. Minimum, desirable, and optimum specifications for precision were calculated using the equations %CV_Minimum < 0.75*CV_I, %CV_Desirable < 0.50*CV_I, and %CV_Optimum < 0.25*CV_I [5] where CV_I is the intra-individual BV reported on the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Biological Variation Database (Table 4) [4]. CV_I estimates were not available for BUN, UA, and HCG. Therefore, analytical performance specifications based on BV could not be calculated for these analytes. Analytes were evaluated against analytical performance specifications for minimum, desirable, and optimum CV and classified as meeting the criteria if all three levels of QC tested met the specification limits. Two analytes (ALB and CA) failed to meet the minimum performance specifications on the Atellica CI Analyzer while three analytes (ALB, CA, and PSA) failed to meet minimum performance specifications on the cobas 6000.

Table 4.

Analytical goals for CV based on biological variation.

Chemistry Analytesa	CVIb	Minimum %CVAPS	Desirable %CVAPS	Optimum %CVAPS	Atellica CI APS Category	cobas 6000 APS Category
ALB	2.5 %	1.9 %	1.3 %	0.6 %	< Minimum	< Minimum
ALP	5.3 %	4.0 %	2.7 %	1.3 %	Desirable	Desirable
ALT	10.1 %	7.6 %	5.1 %	2.5 %	Optimum	Minimum
AMY	6.6 %	5.0 %	3.3 %	1.7 %	Desirable	Desirable
AST	9.6 %	7.2 %	4.8 %	2.4 %	Desirable	Desirable
CA	1.8 %	1.4 %	0.9 %	0.5 %	< Minimum	< Minimum
CHOL	5.3 %	4.0 %	2.7 %	1.3 %	Desirable	Desirable
GLU	5.0 %	3.8 %	2.5 %	1.3 %	Desirable	Minimum
HDLC	5.7 %	4.3 %	2.9 %	1.4 %	Desirable	Minimum
LDLC	8.3 %	6.2 %	4.2 %	2.1 %	Desirable	Desirable
TRIG	19.9 %	14.9 %	10.0 %	5.0 %	Optimum	Optimum

Immunochemistry Analytesa	CVIb	Minimum %CVAPS	Desirable %CVAPS	Optimum %CVAPS	Atellica CI APS Category	cobas 6000 APS Category
FT4	4.9 %	3.7 %	2.5 %	1.2 %	Minimum	Desirable
PSA	6.8 %	5.1 %	3.4 %	1.7 %	Minimum	< Minimum
TSH	17.7 %	13.3 %	8.9 %	4.4 %	Optimum	Optimum

Abbreviations: %CV_APS, analytical performance specification; CV_I, within-subject biological variation

^aCV_I estimates not available for BUN, UA, and HCG

^bCV_I from the European Federation Laboratory Medicine Biological Variation Database

3.2. Method comparison

Passing-Bablok regression demonstrated good concordance for all analytes tested (Fig. 1A and 1Bfig1A). The correlation coefficient (r), slope, y-intercept, and 95 % CIs are summarized in Table 5. Good correlation was observed between the Atellica CI Analyzer and the cobas 6000, with r values for chemistry analytes ranging from 0.989 for ALB and CA to 1.000 for ALP, BUN, and GLU; r values for immunochemistry analytes ranged from 0.980 for FT4 to 0.999 for TSH. Slopes for chemistry analytes ranged from 0.946 for ALB and GLU to 1.143 for AMY while slopes for immunochemistry analytes ranged from 0.798 for FT4 to 1.276 for HCG. A constant bias was present for 10 analytes based on the 95 % CI of the y-intercept not including a value of 0. Bland-Altman plots demonstrating bias across the range of analyte concentrations tested are shown in Fig. 2A, Fig. 2BA–2B. The smallest mean bias observed for chemistry assays was −0.4 % for UA and the largest mean bias observed was 12.4 % for AMY. For immunochemistry assays, the smallest mean bias observed was −3.9 % for TSH and the largest mean bias observed was 26.2 % for HCG.

Fig. 1B.

Bland-Altman plots showing differences between the Atellica CI Analyzer and the cobas 6000 chemistry analytes. Blue dotted lines represent mean bias. Red dashed lines indicate the limits of agreement, defined as the mean difference ±2 times the standard deviation of the differences.

Fig. 1A.

Passing-Bablok regression analysis of chemistry analytes between the Atellica CI Analyzer and the cobas 6000. Red dashed lines represent regression lines and solid black lines represent identity lines.

Table 5.

Summary of Passing-Bablok regression analysis.

Chemistry
Analyte	Units	n	Slope (CI)	Intercept (CI)	r	Sample Range
ALB	g/dL	40	0.946 (0.885, 0.990)	0.318 (0.107, 0.617)	0.989	1.8–5.5
ALP	U/L	40	0.988 (0.975, 1.000)	−0.799 (−1.356, −0.067)	1.000	26–955
ALT	U/L	53	1.060 (1.036, 1.072)	0.896 (0.335, 2.776)	0.998	8–672
AMY	U/L	52	1.143 (1.122, 1.158)	−0.225 (−1.339, 0.803)	0.998	28–1422
AST	U/L	46	1.008 (0.942, 1.036)	−1.489 (−2.623, 0.355)	0.995	11–663
BUN	mg/dL	49	1.006 (0.998, 1.014)	0.620 (0.434, 0.827)	1.000	6.0–108.0
CA	mg/dL	54	1.000 (0.932, 1.068)	−0.219 (−0.763, 0.384)	0.989	4.8–13.4
CHOL	mg/dL	58	0.965 (0.946, 0.987)	2.349 (−1.039, 7.287)	0.998	76–514
GLU	mg/dL	54	0.946 (0.931, 0.960)	2.431 (0.606, 5.065)	1.000	54–543
HDLC	mg/dL	54	1.097 (1.068, 1.127)	−0.228 (−1.474, 1.927)	0.994	5.3–147.3
LDLC	mg/dL	56	0.992 (0.961, 1.021)	8.720 (3.872, 13.47)	0.995	13–340
TRIG	mg/dL	53	0.966 (0.951, 0.977)	6.383 (4.148, 8.559)	0.996	25.0–807.9
UA	mg/dL	42	0.992 (0.981, 1.005)	0.023 (−0.058, 0.081)	0.999	1.4–18.1

Immunochemistry
Analyte	Units	n	Slope (CI)	Intercept (CI)	r	Sample Range
FT4	ng/dL	40	0.798 (0.781, 0.842)	0.184 (0.129, 0.217)	0.980	0.63–6.6
HCG	mIU/mL	40	1.276 (1.192, 1.352)	1.414 (0.259, 6.274)	0.993	9.0–905.0
PSA	ng/mL	40	0.933 (0.878, 0.946)	0.023 (−0.005, 0.079)	0.995	0.90–55.30
TSH	μIU/mL	41	1.023 (1.000, 1.048)	−0.077 (−0.184, −0.007)	0.999	0.110–92.750

Fig. 2A.

Passing-Bablok regression analysis of immunochemistry analytes between the Atellica CI Analyzer and the cobas 6000. Red dashed lines represent regression lines and solid black lines represent identity lines.

Fig. 2B.

Bland-Altman plots showing differences between the Atellica CI Analyzer and the cobas 6000. Blue dotted lines represent mean bias. Red dashed lines indicate the limits of agreement, defined as the mean difference ±2 times the standard deviation of the differences.

4. Discussion

It is estimated that approximately 80 % of patients presenting to the hospital have one or more laboratory tests ordered during their visit [6]. Additional laboratory test requests come from visits to doctor's offices and from other types of healthcare facilities such as nursing homes. Because such testing is used to guide clinical decision making, quality management of the overall testing process is critical to ensure accurate results are provided. As mentioned previously, this includes the verification and/or validation of new laboratory instruments and assays prior to reporting patient results.

Overall, this study demonstrated good analytical performance of the new Atellica CI Analyzer when compared to the cobas 6000. In total, 76 % of the analytes tested on the Atellica CI Analyzer demonstrated within-laboratory CVs of ≤3 % and 100 % of analytes demonstrated within-laboratory CVs ≤4 % at each QC level tested. This is in comparison to the cobas 6000 where ≤3 % within-laboratory CV was observed for 65 % of analytes tested and 82 % of analytes had CVs of ≤4.0 % across all levels of QC tested. Repeatability and within-lab precision were evaluated using QC material which can have a sample matrix that differs from native samples. However, evaluation of precision using pooled patient samples (one replicate per day for five days) demonstrated comparable performance to that observed with QC material.

There are three widely accepted approaches or models to establishing quality goals for laboratory tests to ensure results are fit for the clinical purpose for which they will be used [7]. Quality goals can be based on (1) the effect of analytical performance on clinical outcomes, (2) components of biological variation of the analyte, or (3) state-of-the-art [7,8]. Here, performance of assays on the Atellica CI Analyzer was compared against both the manufacturer specifications and against criteria established based on BV data. It has been previously reported that meeting analytical performance specifications based on BV is difficult for analytes that are under strict physiological controls and have a narrow range [1,[9], [10], [11]]. It is not surprising then that ALB and CA failed to meet even the minimum criteria for analytical performance goals when using BV on either platform. Using extremely stringent analytical goals for some assays such as these may not be appropriate and it has been suggested that medical outcome and need also be considered when setting criteria for analytical performance specifications [12].

Overall, method comparisons between the Atellica CI Analyzer and the cobas 6000 were acceptable. It is not unusual to observe biases between assays produced by different manufacturers due to differences in assay methodology, instrumentation, proprietary epitope specific antibody (Ab) designs, unknown isoform detection, assay matrix effects, and analytical interferents. Differences in metrological traceability of assays can also impact comparability of results across vendors. Table 6 summarizes the metrological traceability of Atellica and cobas assays used in this study, as reported in their respective IFUs. Interestingly, although the Atellica IM and cobas 6000 assays for measuring HCG are traceable to the same World Health Organization (WHO) 4th IS 75/589 reference material, a mean bias of approximately 26 % was observed between the methods. Differences in analytical specificity leading to variability between HCG assays has been reported previously and largely attributed to the molecular heterogeneity of HCG that exists due to multiple HCG isoforms, differences in glycosylation, and carboxyl-terminal variants [13,14]. These differences in how immunoassays recognize various isoforms of HCG suggests non-commutability of the reference material [15,16]. Understanding such biases when changing methods in the laboratory and the potential impact to reference intervals can help facilitate appropriate communication with clinicians [17].

Table 6.

Traceability of chemistry and immunochemistry assays used in method comparison studies.

Chemistry Analyte	Siemens Atellica CI Analyzer	cobas 6000 Analyzer
ALB	BCG reference method, which uses SRM 927 reference materials from the NIST.	Reference preparation of the IRMM BCR470/CRM470 (RPPHS)
ALP	Primary reference procedure for the measurement of catalytic activity of alkaline phosphatase at 37 °C as described by the IFCC.	IFCC procedure (2011)
ALT	IFCC reference method, which uses IFCC-454.	Original IFCC formulation using calibrated pipettes together with a manual photometer providing absolute values and the substrate-specific absorptivity, ε.
AMY	IRMM/IFCC-456 reference material	Roche system reagent using calibrated pipettes together with a manual photometer providing absolute values and substrate-specific absorptivity, ε.
AST	IFCC reference method, which uses ERM-AD457/IFCC.	Original IFCC formulation using calibrated pipettes together with a manual photometer providing absolute values and the substrate-specific absorptivity, ε.
BUN	CDC reference method, which uses SRM 912 and 909 reference materials from NIST.	Reference material SRM 912 from NIST.
CA	NIST atomic absorption reference method, which uses SRM 915 and SRM 909b reference materials from NIST.	SRM 956 c Level 2 reference material
CHOL	NCEP/CDC reference method, which uses SRM 909 reference materials from NIST.	Abell/Kendall and ID/MS.
GLU	SRM 965a from NIST.	ID/MS
HDLC	Designated CDC reference method (ultracentrifugation method)	Designated CDC reference method (ultracentrifugation method)
LDLC	NCEP beta-quantification reference method for LDL-cholesterol.	NCEP beta-quantification reference method for LDL-Cholesterol.
TRIG	Reference material SRM909 from NIST.	ID/MS
UA	CDC candidate reference method, which uses SRM 913 and SRM 909 reference materials from NIST.	ID/MS

Immunochemistry Analyte	Siemens Atellica CI Analyzer	cobas 6000 Analyzer
FT4	Internal standard manufactured using U.S.P. material	The Elecsys FT4 IV assay was standardized against the Elecsys FT4 III method. The Elecsys FT4 III assay is traceable to the Elecsys FT4 II assay which in turn is traceable to the Enzymun-Test FT4 which has been standardized using equilibrium dialysis.
HCG	WHO 4th IS for Chorionic Gonadotropin, Human (IRP 75/589)	The 4th IS for Chorionic Gonadotropin, Human, code 75/589
PSA	WHO 1st IS for PSA (90:10) (IRP 96/670).	WHO 1st IS for PSA (90:10) (IRP 96/670).
TSH	WHO 3rd IS for human TSH (IRP 81/565).	2nd IRP WHO Reference Standard 80/558

BCG, Bromocresol Green; CDC, United States Centers for Disease Control; ID/MS, isotope dilution/mass spectrometry; IFCC, International Federation of Clinical Chemistry; IRMM, Institute for Reference Materials and Measurement; IS, International Standard; NCEP, National Cholesterol Education Program; NIST, National institute of Standards and Technology; RPPHS, Reference Preparation for Proteins in Human Serum; SRM, standard reference material; U.S.P., United States Pharmacopeia; WHO, World Health Organization.

One limitation of this study is that it did not verify reference intervals for each of the assays tested which should be completed prior to reporting patient results. However, the strong analytical performance of the Atellica CI Analyzer confirms the suitability of the new instrument for routine clinical use. Additional studies are needed to evaluate potential efficiencies for laboratory workflow that might be realized with the implementation of this new system.

Funding

This study was funded by Siemens Healthcare Diagnostics Inc., Tarrytown, NY.

CRediT authorship contribution statement

Ulla Ruffing: Writing – original draft, Project administration. Sabrina Mickeler: Methodology, Investigation. Michaela Kraft: Methodology, Investigation. Peter Findeisen: Writing – review & editing, Data curation, Conceptualization.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:

Dr. Peter Findeisen reports article publishing charges, equipment, drugs, or supplies, and writing assistance were provided by Siemens Healthineers AG. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

Data will be made available on request.