Accuracy of pulse CO-oximetry to evaluate blood carboxyhemoglobin level: a systematic review and meta-analysis of diagnostic test accuracy studies

Abstract

Carbon monoxide (CO) poisoning is one of the most common causes of poisoning death and its diagnosis requires an elevated carboxyhemoglobin (COHb) level. Noninvasive CO saturation by pulse oximetry (SpCO) has been available since 2005 and has the advantage of being portable and easy to use, but its accuracy in determining blood COHb level is controversial. To evaluate the accuracy of SpCO (index test) to estimate COHb (reference test). Systematic review and meta-analysis of diagnostic test accuracy (DTA) studies. Four electronic databases were searched (Medline, Embase, Cochrane Central Register of Controlled Trials, and OpenGrey) on 2 August 2022. All studies of all designs published since the 2000s evaluating the accuracy and reliability of SpCO measurement compared to blood COHb levels in human volunteers or ill patients, including children, were included. The primary outcome was to assess the diagnostic accuracy of SpCO for estimating COHb by blood sampling by modeling receiver operating characteristic (ROC) curves and calculating sensitivity and specificity (primary measures). The secondary measures were to calculate the limits of agreement (LOA) and the mean bias. This systematic review was conducted according to the Preferred Reporting Items for a Systematic Review and Meta-analysis-DTA 2018 guidelines and has been registered on International Prospective Register of Systematic Reviews (PROSPERO, CRD42020177940). The risk of bias was evaluated using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. Twenty-one studies were eligible for the systematic review; 11 could be included for the quantitative analysis of the primary measures and 18 for the secondary measures. No publication bias was found. The area under the summary ROC curve was equal to 86%. The mean sensitivity and specificity were 0.77, 95% confidence interval (CI, 0.66–0.85) and 0.83, 95% CI (0.74–0.89), respectively (2089 subjects and 3381 observations). The mean bias was 0.75% and the LOA was −7.08% to 8.57%, 95% CI (−8.89 to 10.38) (2794 subjects and 4646 observations). Noninvasive measurement of COHb (SpCO) using current pulse CO oximeters do not seem to be highly accurate to estimate blood COHb (moderate sensitivity and specificity, large LOA). They should probably not be used to confirm (rule-in) or exclude (rule-out) CO poisoning with certainty.

Introduction

Carbon monoxide (CO) poisoning is one of the most common causes of poisoning death and is responsible for approximately 50 000 emergency department (ED) visits per year in the USA [1]. In 2014 in the USA, 1319 deaths from accidental or intentional CO poisoning were reported, and a decline in annual cases has been seen since 1999 [2]. Worldwide, the estimated incidence of CO poisoning is 137 cases per million and 4.6 deaths per million [3]. However, the numbers of intoxication cases are most likely underestimated [4,5]. Carbon monoxide is an odorless and colorless gas, and the symptoms of CO poisoning are nonspecific: headache, dizziness, nausea, vomiting, fatigue, and loss of consciousness. The main sources of intoxication are fires, heating defects, and cars [6]. The diagnosis of CO poisoning requires a history of recent CO exposure, the presence of symptoms, and an elevated carboxyhemoglobin (COHb) level [7]. Either arterial or venous blood may be used to measure the COHb level [7–9]. Since 2005, noninvasive pulse CO oximeters have been available and approved by the Food and Drug Administration (FDA). They can quickly estimate COHb using readings at eight wavelengths of light at the fingertip (SpCO, CO saturation by pulse oximetry) [10,11]. As early as 2006, protocols were published on the use of noninvasive pulse CO oximeters for the triage of the victims of suspected CO poisoning directly on the scene by first responders [12]. Actually, CO oximeters are used daily by thousands of fire and emergency medical services around the world [13]. Nevertheless, the results of studies evaluating the accuracy of SpCO in assessing COHb were heterogeneous and brought into question its use in clinical practice [7,11]. Some experts do not recommend its use for the diagnosis of patients suspected of having CO poisoning in the ED [10].

The aim of this systematic review and meta-analysis was to evaluate the accuracy of SpCO (index test) to estimate COHb in any population.

Methods

This systematic review was conducted according to the Preferred Reporting Items for a Systematic Review and Meta-analysis-DTA 2018 guidelines [14] and has been registered on International Prospective Register of Systematic Reviews (PROSPERO, CRD42020177940).

Population

We included studies evaluating the accuracy and reliability of multiple wavelength pulse CO-oximeter (SpCO) measurements (index test) in human volunteers or ill patients, including children, compared to blood COHb levels (reference test). Studies of all designs published after 2000 were eligible, without language restrictions. Case reports were not included.

Search strategy, information sources, and study selection

Two authors (F.J. and C.L.) independently screened titles, abstracts, and full texts, following the inclusion criteria. The discrepancies (n = 6) were resolved after a discussion between the two authors (and if necessary, the intervention of a third author, M.P.; n = 0). We performed searches of the following databases on 2 August 2022: Medline (via Pubmed), Embase, and Cochrane Central Register of Controlled Trials (CENTRAL) on the Cochrane Library. We also searched the grey literature via OpenGrey. Our search algorithms for different databases included the following terms: (‘oximetry’, ‘pulse oximetry’, ‘CO oximeter’, ‘CO oximetry’, ‘pulse CO oximetry’, ‘noninvasive’) and (‘carboxyhemoglobin’, ‘COHb’). Quantitative data from the studies were independently extracted by two authors (M.P. and F.J.).

Quality assessment

The risk of bias was independently evaluated by two authors (M.P. and C.L.) using the Quality Assessment of Diagnostic Accuracy Studies-2 tool [15]. The assessed risks of bias were selection bias, studied test use, reference test use, and timing between the tests. Each of these items allowed the study to be classified into different risk categories (low, high, or unclear). In cases of discordance, the classification was discussed between the authors. All discrepancies were resolved by consensus (n = 2).

Study outcomes

The primary outcome was to estimate the diagnostic accuracy by modeling receiver operating characteristic (ROC) curves and measuring sensitivity (Se) and specificity (Sp) of SpCO for estimating COHb by blood sampling (primary measures). The secondary measures were to report the limits of agreement (LOA), the mean bias, and to perform subgroup analyses: according to the SpCO device used, and among subjects with suspected CO poisoning.

Data analysis

The analyses were performed by a biostatistician (B.L.) using R software version 4.0.3 (The R Foundation, Boston, Massachusetts, USA) (2020-10-10). The meta-analysis was performed using the mada package version 0.5.10 (mada: Meta-Analysis of Diagnostic Accuracy, https://cran.r-project.org/web/packages/mada), focusing on Se and Sp measures. The mean Se and Sp, and their 95% confidence regions were estimated using a bivariate linear mixed model that was proposed in 2005 by Reitsma et al [16]. In the absence of a definition of the threshold of positivity, it was set at ≥10% (SpCO and COHb) when individual data were available. Summary ROC curves (SROC) were estimated using three methods: Rutter and Gatsonis [17], Moses et al. [18], and Rücker and Schumacher [19]. LOA calculations were based on the method described by Tipton and Shuster [20]. The analysis of publication bias was performed using a Deeks’ funnel plot and its significance was tested using the associated test. This method is recommended for the evaluation of diagnostic studies [21].

Results

Study selection

Of the 293 reports identified from databases and 6 from citation searching and websites, 21 studies were included in the systematic review and 11 in the meta-analysis for the primary measures (absence of exploitable quantitative data, n = 10), and 18 for the secondary measures (absence of exploitable quantitative data, n = 3) (Fig. 1).

Fig. 1.

Flow diagram of study selection. COHb, carboxyhemoglobin; SpCO, pulse carbon monoxide oximetry.

Study characteristics

Of the 21 articles included [22–42], 5 studies were experimental and 16 observational (Table 1). The experimental studies consisted of breathing CO in healthy volunteers and performing several measurements of SpCO and COHb. Among the observational studies, 10 were conducted in the ED and 2 in a hyperbaric center. Three different devices were used to measure SpCO: Rad-57 (Masimo, Irvine, California, USA) in 14 studies, Radical-7 (Masimo, Irvine) in 6 studies, and V-Spec Monitoring System (Senspec, Rostock, Germany) used once, in an experimental study.

Table 1.

Main characteristics of the studies included in the systematic review (n = 21)

Study	Design	Clinical setting	Number of subjects	Number of observations	Participant characteristics	SpCO device (index test)	COP SpCO threshold	COHb measurement (reference test), sample size, and devices	COP COHb threshold	Funding and COI	Inclusion in the meta-analysis for primary (SROC, Se, Sp) or secondary (LOA, mean bias) measures
Barker et al. (2006) [22]	Experimental	NA	10	530	Healthy adults	RAD-57 (Masimo)	NA	Arterial blood ABL730 (Radiometer, Københav, Denmark) and OSM-3s (Radiometer)	NA	Masimo and Department of Anesthesiology, University of Arizona College of Medicine	Secondary
Bidstrup et al. (2021) [23]	Observational	Hyperbaric center	10	50	COP adults treated with HBO2	Radical-7 (Masimo)	NA	Arterial or venous blood ABL800 FLEX (Radiometer)	NA	Loan of equipment (device, sensors, computer) by Masimo	Secondary
Caboot et al. (2012) [24]	Observational	Comprehensive sickle cell center	50	50	Children with sickle cell disease	Radical-7 (Masimo)	NA	Arterial or venous blood RapidLab 1265 (Siemens, Huntersville, North Carolina, USA)	NA	Supported by the National Heart, Lung, and Blood Institute, and the National Center for Research Resources	Secondary
Cardwell et al. (2012) [25]	Observational	Operating room	203	203	Preanesthetic children	Radical-7 (Masimo)	NA	Venous blood RapidLab C1265 (Siemens)	NA	Supported by NIH/NCRR Colorado CTSI The pulse CO-oximeters were purchased by Children’s Hospital Colorado The reusable finger sensors and SpCO software were loaned by Masimo	Secondary
Coulange et al. (2008) [26]	Observational	ED	12	12	Suspected COP nonsmoking patients	RAD-57 (Masimo)	≥10% (individual data)	Venous blood IL 682 CO-oximeter (Instrumentation Laboratory, Barcelona, Spain)	≥10% (individual data)	None	Primary Secondary
Feiner et al. (2013) [27]	Experimental	NA	12	283	Healthy nonsmoking adults breathing CO gas to produce a target %COHb level of ≈10% to 12%	Radical-7 (Masimo)	≥6.6%	Arterial blood ABL800 FLEX (Radiometer)	≥10%	Masimo	Primary Secondary
Kaya et al. (2018) [28]	Observational	ED	213	213	Suspected COP adults	RAD-57 (Masimo)	>23%	Arterial or venous blood ABL700 (Radiometer)	>23%	None	Secondary
Kot et al. (2008) [29]	Observational	Hyperbaric center	49	49	Suspected COP patients	RAD-57 (Masimo)	≥20%	Capillary blood CO-xylite AVL 912 (AVL Biomedical Instrument, Basle, Switzerland)	≥20%	None	Primary Secondary
Koyuncu et al. (2020) [30]	Observational	ED	106	106	Suspected COP adults	RAD-57 (Masimo)	NA	Arterial blood	>5% for nonsmokers >10% for smokers	None	Secondary
Kulcke et al. (2016) [31]	Experimental	NA	6	888	Healthy nonsmoking adults	V-Spec (Senspec)	≥8.9%	Arterial blood ABL90 FLEX (Radiometer) or OSM3, (Radiometer)	≥10%	University of California, San Francisco, Department of Anesthesia and Perioperative Care. Oximeter and probes were supplied by Senspec	Primary Secondary
Nagano et al. (2008) [32]	Experimental	NA	10	10	Healthy smoking adults	RAD-57 (Masimo)	NA	Sample blood (not specified)	NA	None	Absence of exploitable data for the meta-analysis
Piatkowski et al. (2009) [33]	Observational	Burn center	20	20	Suspected COP patients	RAD-57 (Masimo)	≥10% (individual data)	Venous blood ABL700 (Radiometer)	≥10% (individual data)	Pulse CO-oximeter device was supplied by Masimo	Primary Secondary
Roth et al. (2011) [34]	Observational	ED	1578	1578	All ED patients	Radical-7 (Masimo)	>6.6%	Arterial or venous blood ABL700 (Radiometer)	>6.6%	Masimo	Primary Secondary
Ruppel et al. (2011) [35]	Observational	Pulmonary function laboratory	139	139	Patients referred for DLCO and arterial blood gas measurement.	RAD-57 (Masimo)	NA	Arterial blood RapidPoint 405 (Siemens)	NA	None	Secondary
Sebbane et al. (2013) [36]	Observational	ED	93	93	Suspected COP adults	RAD-57 (Masimo)	≥9%	Venous blood IL 682 CO-oximeter (Instrumentation Laboratory, Milan, Italy)	>5% for nonsmokers >10% for smokers	None	Primary Secondary
Suner et al. (2008) [37]	Observational	ED	64	64	All ED patients ≥17 years old	RAD-57 (Masimo)	≥9% for nonsmokers ≥13% for smokers	Venous blood	≥9% for nonsmokers ≥13% for smokers	Masimo loaned the devices and supported the data collection by providing funds for research assistants	Primary
Touger et al. (2010) [38]	Observational	ED	120	120	Suspected COP patients	RAD-57 (Masimo)	≥15%	Arterial or venous blood RapidLab 1200 (Siemens)	≥15%	Devices and training were supplied by Masimo	Primary Secondary
Villalba et al. (2019) [39]	Observational	ED	126	126	Suspected COP patients with SpCO ≥10%	Radical-7 (Masimo)	≥10%	Venous blood XN9000 hematology analyzer (Sysmex America, Lincolnshire, Illinois, USA)	≥10%	Masimo funding for the technicians and the administrative costs; loaning of the SpCO sensors and equipment; and data safety monitoring COI: the authors have disclosed a relationship with Masimo	Primary Secondary
Weaver et al. (2013) [40]	Observational	ED	1363	1363	All ED patients	RAD-57 (Masimo)	>6%	Arterial or venous blood ABL825 (Radiometer)	>6%	Centers for Disease Control and SciMetrika. Masimo provided the oximeters and educated the research staff COI: the authors have disclosed relationships with SciMetrika and Masimo	Absence of exploitable data for the meta-analysis
Zaouter et al. (2012) [41]	Experimental	NA	9	148	Healthy nonsmoking adults	RAD-57 (Masimo)	≥10%	Venous blood ABL80 FLEX (Radiometer)	≥10%	None	Primary Secondary
Zorbalar et al. (2014) [42]	Observational	ED	38	38	Headache patients with SpCO ≥10% and ≥14 years old	RAD-57 (Masimo)	>10%	Venous blood GEM Premier 3000-GEM OPL CO-Oximeter (Instrumentation Laboratory)	>10%	Masimo loaned the devices and sensors	Secondary

COHb, carboxyhemoglobin; COI, conflicts of interest; COP, carbon monoxide poisoning; DLCO, diffusing capacity of the lungs for carbon monoxide; ED, emergency department; HBO₂, Hyperbaric hyperbaric oxygen; LOA, limits of agreement; NA, not applicable; ROC, receiver operating characteristic; Se, sensitivity; Sp, specificity; SpCO, pulse carbon monoxide oximetry; SROC, summary ROC curves.

Risk of bias and concerns

The patient selection was the principal source of bias and concerns regarding applicability (Table 2). Indeed, the index test was mostly well documented such as the reference one. Besides, the concerns regarding applicability were low because the threshold was predefined, and the biological result was not subject to interpretation. However, the patient selection was less clear in some of the studies. We found that in 33% of the studies, patient selection could have introduced a high risk of bias, and in 33%, the risk of bias induced by patient selection is doubtful. Obviously, this is leading to an increase in concerns regarding the applicability of the patient selection. To a lesser extent, the flow and timing were also unclear, with 28% of studies with a high risk of bias induced by the flow and timing design.

Fig. 2.

Sensitivities and specificities of the 11 included studies.

Table 2.

Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2)

Masimo was involved in the funding or loan of materials in 11 of 20 of the studies evaluating their devices (RAD-57 and Radical-7), and Senspec also provided materials for the study evaluating the V-Spec Monitoring System.

Outcomes of individual studies

The observed sensitivity values were very different between studies, ranging from 48% for Touger et al. [38] to 100% for Piatkowski et al. [33], Roth et al. in nonsmokers subgroup [34], and Sebbanne et al. in smokers subgroup [36]. The same finding was observed for specificity ranging from 51% for Feiner et al. when the COHb threshold was ≥5% [27] to 99% for Touger et al. (threshold ≥ 15%) [38]. The highest area under the ROC curve was 99% (Feiner et al.) when a COHb threshold ≥5% was used to define positive cases [27]. The LOA varied from ±3% for Kulcke et al. [31] to ±15% (calculated on individual data) for Zorbalar et al [42].

Primary measures

A total of 11 studies were included to estimate SROC and mean Se and Sp (10 studies without exploitable data for the primary measures). Overall, 2089 subjects and 3381 observations were analyzed (i.e. coupled measurement of SpCO and COHb). The mean Se and Sp were 0.77, 95% CI (0.66–0.85) and 0.83, 95% CI (0.74–0.89), respectively (Fig. 2). The area under the SROC curve, estimated using the Rutter and Gatsonis method, was equal to 86%. To achieve a Se of 95% the Sp was between 45 and 50%, depending on the method used. Conversely, for a Sp of 95%, the Se was between 42 and 55% (Fig. 3).

Fig. 3.

Summary ROC curves from the 11 included studies. ROC, receiver operating characteristic.

Although no publication bias was statistically found, with an estimated bias of −6.37 (not significantly different from zero, P = 0.22), there was an asymmetry on the funnel plot concerning only studies with small sample sizes (Fig. 4).

Fig. 4.

Funnel plot of the 11 included studies.

Secondary measures

A total of 18 studies were included to estimate the LOA (3 studies without exploitable data for the LOA). Overall, 2794 subjects and 4646 observations were analyzed. The mean bias was 0.75%, 95% CI (−6.26 to 7.75) and the LOA was −7.08% to 8.57%, 95% CI (−8.89 to 10.38) (Fig. 5).

Fig. 5.

Limits of agreement of the 18 included studies for the secondary measures.

When the device used was the Rad-57, the mean Se and Sp were 0.70, 95% CI (0.56–0.81) and 0.88, 95% CI (0.77–0.94) (Figure E1 in the online data supplement, Supplemental Digital Content 1, http://links.lww.com/EJEM/A377), respectively; and the LOA was −7.83% to 8.73%, 95% CI (−10.09 to 10.99) (mean bias, 0.45%; Figure E2 in the online data supplement, Supplemental Digital Content 1, http://links.lww.com/EJEM/A377). The mean Se and Sp were 0.74, 95% CI (0.59–0.85) and 0.79, 95% CI (0.73–0.85) when using the Radical-7 device (Figure E3 in the online data supplement, Supplemental Digital Content 1, http://links.lww.com/EJEM/A377), respectively; and the LOA was −5.89% to 8.45%, 95% CI (−9.74 to 12.30) (mean bias, 1.28%; Figure E4 in the online data supplement, Supplemental Digital Content 1, http://links.lww.com/EJEM/A377). Among patients with suspected CO poisoning the mean Se and Sp was 0.70, 95% CI (0.56–0.81) and 0.88, 95% CI (0.77–0.94) (Figure E5 in the online data supplement, Supplemental Digital Content 1, http://links.lww.com/EJEM/A377), respectively; and the LOA was −8.11% to 8.59%, 95% CI (−10.74 to 12.23) (mean bias, 0.74%; Figure E6 in the online data supplement, Supplemental Digital Content 1, http://links.lww.com/EJEM/A377).

Discussion

In this meta-analysis, we found that the SpCO level had a mean Se of 77% and a mean Sp of 83% to determine CO poisoning using COHb as reference. The LOA of SpCO level was ±8% (mean bias +1%). Some experts have suggested by consensus that acceptable LOA values in this context are ±5% [6,38]. Nevertheless, the FDA 510(k) accuracy specification of the RAD-57 is 3% (SD) corresponding to LOA of ±5.9% [13]. Therefore, the noninvasive measurement of COHb is clearly less accurate than advertised by the manufacturers. Indeed, here we found LOA of ±8.3% when the Rad-57 was used.

The use of CO oximeters for prehospital triage can save time for admission to the hyperbaric center and limit the number of hospital transfers [43]. In clinical practice, SpCO level has a major influence on the decision to transfer, or not, suspected CO poisoning victims to hospitals [44]. However, to achieve a safety rule-out the test used must have a high Se [45]. Here, we found a false-negative rate (1 – Se) of 23% with the SpCO measurement, considering the blood COHb level as reference. By selecting only studies with subjects with suspected CO poisoning, up to 30% of false-negatives were found. There were even negative SpCO levels (0%) while COHb was very high (>35%) [38]. The use of CO oximeters for the triage of potential CO poisoning victims is already discouraged by several experts because of their lack of accuracy, and the results from our meta-analysis also suggest this [6,10,11].

The CO oximeters are proposed as a screening tool in the ED in populations with nonspecific symptoms, to detect occult CO poisoning [10]. Among the three studies included in our systematic review evaluating this population (unselected cohorts in the ED), the false-positive rates were 9%, 23%, and 54% [34,37,40]. A study started in an urban ED on all admitted patients was prematurely stopped because all patients with positive SpCO (>8% in nonsmokers and >13% in smokers, n = 5) had negative blood COHb levels. Emergency staff became skeptical about the clinical usefulness of these devices and decided to stop this research [46]. In addition, an incidental finding of elevated COHb does not always correlate with CO poisoning [47]. Indeed, it can be due to endogenous production of CO, as found in hemolytic anemia [48]. The use of CO oximeters in an unselected population in the ED would likely be responsible for misdiagnoses as well as an increase in unnecessary COHb blood tests.

In our meta-analysis, the positive likelihood ratios (LR+) was 4.5 and the negative LR (LR−) was 0.3. Usually, LR+ >10 and LR− < 0.1 are considered to provide strong evidence to admit (rule-in) or exclude (rule-out) diagnoses, respectively [49,50]. With LR+ between 2 and 5 and LR− between 0.2 and 0.5, the contribution of SpCO in determining blood COHb level is considered low [51].

A possible alternative to CO oximeters is the measurement of exhaled CO, but the correlation with blood COHb level is uncertain and the measurement can be biased, especially in patients with severe airflow obstruction [52–55]. The correlation between capillary and venous COHb seems to be very high and could be assayed as part of triage using point-of-care analyzers [56].

Limitations

First, we included heterogeneous populations in our analysis, which may have introduced noise into our model. Nevertheless, the whole population is concerned by CO poisoning, and for this reason, we did not exclude healthy subjects, sick subjects, or children.

Second, the studies included in the meta-analysis did not use the same COHb thresholds to define CO poisoning (from >5% to >20%); sometimes without distinction between smokers and nonsmokers. The same is true for the SpCO thresholds used, which also varied from >5% to >20%. It is conventionally accepted that COHb values >3–4% in nonsmokers and >10% in smokers are considered pathological [8]. The use of these different thresholds has an influence on the calculated Se and Sp values. In addition, in most studies, SpCO measurements were performed in optimal settings, and subjects with nail pathologies or nail polish were excluded, which is a deviation from current practice and tended to overestimate the Se and Sp.

Third, we were unable to obtain the individual patient-level data from all the studies and create a global Bland-Altman plot. Furthermore, some studies could not be included in the quantitative analysis due to the lack of usable results. Indeed, diagnostic studies should clearly report the 2 × 2 diagnostic contingency matrix to be properly exploitable [57]. Therefore, there are only a limited number of studies included in this systematic review, and also a small number of subjects.

Conclusion

Noninvasive measurement of COHb (SpCO) using current pulse CO-oximeters does not seem to be highly accurate to estimate blood COHb. In clinical practice, it does not appear possible to replace them with blood tests either for safely ruling in or ruling out CO poisoning. Future studies are needed to assess whether SpCO measurement has an impact on outcomes or provides any benefit in the management of subjects with suspected CO poisoning.

Acknowledgements

The authors thank Dr Oussama Gasmi (Department of Paediatric Emergency Medicine, Nantes University Hospital, France) for his help in structuring the database.

F.J., C.L., and B.L. conceived and designed the study. F.J. and C.L. conducted the literature search and selected the studies. M.P. and C.L. analyzed the quality of the studies. M.P. and F.J. collected the data. B.L. analyzed the data, B.L. and F.J. interpreted the data. M.P. and F.J. wrote the first draft of the article, with all other authors making important critical revisions. All authors have read and approved the final version of the article.

Conflicts of interest

F.J. is a member of the editorial board of this journal. For the remaining authors, there are no conflicts of interest.