A test of validity of a new open-circuit indirect calorimeter

Abstract

Background

Indirect calorimetry is an accurate way to measure resting metabolic rate. The Deltatrac^® Metabolic Monitor is considered a criterion standard but is no longer manufactured. New-generation indirect calorimeters have been introduced, but there is limited published validation data comparing these devices to criterion instruments.

Materials and Methods

A prospective, observational, n-of-1 trial was conducted to validate a new-generation indirect calorimeter against a gold standard device. This design was chosen in order to minimize and define the degree of biological variation, thus focusing on variation due to the devices. Measurements of gas exchange using both indirect calorimeters were conducted daily for 10 consecutive days. Another set of measurement pairs was conducted using just the criterion device for 10 days. Ninety-five percent confidence intervals of differences were used to test for bias. Precision was defined as repeat measures with one device falling within 5% of the other at least 90% of the time.

Results

There were no statistically significant differences between the devices for any measured or calculated parameter. Inter-device differences were no larger than intra-device differences using the criterion instrument. The values obtained from the new device were precise and unbiased compared to the values obtained from the gold standard device.

Conclusion

The new indirect calorimeter measures gas exchange in a reliable and accurate manner compared to a gold standard device. The two devices are equivalent.

Clinical Relevancy Statement

The current gold standard open-circuit indirect calorimeter device, the Deltatrac^® Metabolic Monitor, is no longer being manufactured. Several new-generation indirect calorimeters have been introduced, but there is limited published validation data for these devices. Thus, uncertainty exists as to whether measurements with these new devices are equivalent to measurements with the Deltatrac^®. In an n-of-1 trial, measurements from a QuarkRMR^® were compared to those from a Deltatrac.^® The QuarkRMR,^® in spontaneous breathing mode, was found to be precise and unbiased, giving equivalent values for resting metabolic rate in spontaneous breathing mode.

Introduction

Assessment of total energy expenditure is one of the fundamental functions performed during nutrition assessment. Resting metabolic rate is the largest component of total energy expenditure. Both predictive equations and indirect calorimetry measurements are used to determine resting metabolic rate but the most accurate method is indirect calorimetry.

The Deltatrac^® Metabolic Monitor has become acknowledged as a gold standard among indirect calorimetry devices through several validation studies (1,2) and years of use in clinical and research settings. Production of the Deltatrac^® has been discontinued, but a number of new devices have been introduced into the market that potentially could serve as a replacement. There have been a few attempts to assess the validity of these instruments, and the results have been mixed (3–5). The purpose of the current study was to examine the precision, bias, and reliability of one new indirect calorimeter, the QuarkRMR^®, in comparison to the Deltatrac^®.

Methods

A form of “N-of-1” methodology (6,7) was used to validate the QuarkRMR^® metabolic monitor (Cosmed, Rome, Italy) against a criterion method, the Deltatrac^® Metabolic Monitor (Sensormedics, Yorba Linda, CA). The intent was to repeatedly test both devices in a single subject in order to both define and minimize biological variation, thereby focusing on variation due to the devices. The current study was approved by the institutional review board at our institution. Informed consent was obtained from the subject.

Study Procedure

The subject (one of the authors) reported at 0600 having fasted and avoided vigorous exercise for the previous 10 hours. Consumption of calorie-containing beverages, caffeine, or nicotine were not allowed (8), but the subject was permitted to have sips of water two hours prior to testing. Upon entering the testing room, the subject lay supine on a cot and rested for 30 minutes before any testing was started. The subject did not get up from the cot until all testing was completed for that day and did not move during or between measurements. This subject was familiar with indirect calorimetry equipment and procedures, having operated indirect calorimeters extensively and having been measured on four previous occasions. One of the authors operated both devices in all the test sessions. This operator had four years of experience using indirect calorimeters in critically ill, acutely ill, and healthy individuals.

Data were collected in three phases. In Phase 1, five test sessions were conducted over five consecutive days. Each session consisted of two measurements using the Deltatrac^® performed with a one-minute space between the two measurements. The gas collection hood was removed between the measurements, and the subject did not rise. The purpose of this phase was to measure the extent of variation within repeated Deltatrac^® measurements (intra-device) while minimizing variation due to the subject. This phase also provided data for a power analysis for the main part of the experiment, which was Phase 2. This stage consisted of 10 sessions spaced 24 hours apart. Each session consisted of three measurements applied randomly (QuarkRMR^®-Deltatrac^®-QuarkRMR^® vs. Deltatrac^®-QuarkRMR^®-Deltatrac^®). This phase was designed to test the variation due to the QuarkRMR^® relative to the Deltatrac^® (inter-device) while minimizing variation due to the subject. Phase 3 was a repeat of Phase 1, in which only the Deltatrac^® was used for two measurements with one minute between each. The purpose of this phase was to increase the number of intra-device observations of variation due to the Deltatrac^® and test subject alone. These five sessions were conducted 24 hours apart.

All tests were conducted in a private room in which the ambient temperature was 20.6 degrees centigrade. Blankets were used to keep the subject warm as 20.6 degrees is slightly outside the range of thermoneutrality (9). The room had windows that provided the only light in the room. Besides the sound of the devices there was no other ambient noise.

Indirect Calorimetry Protocol

Each measurement consisted of a 30-minute gas collection period. Clear plastic canopies were used for collection of exhaled gas for both devices. The first five minutes of every test were discarded. The coefficient of variation for oxygen consumption and carbon dioxide production of the remaining 25 minutes of study time had to be ≤10% to be considered steady state. If this coefficient of variation was not achieved, the measurement output was visually searched for the first 10-minute period in which a coefficient of variation ≤10% for oxygen consumption and carbon dioxide production was observed (8,10).

Both devices were warmed up and calibrated according to manufacturer instructions before each test session. For calibration of the gas sensors of the Deltatrac^® a gas mixture of 96% oxygen and 4% carbon dioxide was used. The QuarkRMR^® was calibrated with a gas mixture of 16% oxygen, 5% carbon dioxide, balance nitrogen. In addition to the gas calibration with standard gasses, a calibration against room air was also conducted in the QuarkRMR^®. Both of the devices utilize paramagnetic sensors to measure oxygen concentrations and infrared sensors to measure carbon dioxide concentrations. Expired volume was measured in the Deltatrac^® using a dilution method in which room air is mixed with the expired air to a constant volume of 39 L/min. The QuarkRMR^® measured expired gas volume with a bidirectional digital turbine. The turbine was calibrated using a 3-liter syringe before each session.

Statistics

The intent of the double crossover design of Phase 2 (three measurements with the first and third from the same device) was to determine if differences existed in the gas exchange measurements over time. If the time effect was minimal, the first and third measurements, being made with the same device, were to be averaged to arrive at the mean value for that device during that session. If a time factor independent of device was detected, the results would be reported but not included in any further analysis.

The Anderson-Darling statistic was applied to the oxygen consumption and carbon dioxide production values to determine whether these variables were normally distributed. The Student’s paired t-test was used to analyze the differences in oxygen consumption, carbon dioxide production, respiratory quotient, and resting metabolic rate between the two devices. Bias of the QuarkRMR^® relative to the Deltatrac^® in Phase 2 and of the intra-device Deltatrac^® measurements in Phase 1 and 3 was determined by calculating the 95% confidence interval of the differences between the pairs of measurements. Ninety-five percent confidence intervals that excluded zero indicated bias. Precision of the QuarkRMR^® was defined as at least 90% of measurements falling within 5% of their Deltatrac^® counterparts.

Power analysis

The mean variability in resting metabolic rate between two measurements by the Deltatrac^® in the same subject (Phase 1) was 1.4% and the maximum difference was 2.8%. The standard deviation was 1.8%, equating to a 31 kcal/day difference between the first and second test for this subject. Choosing 31 kcal as the difference to detect, and based on the measured variability, ten inter-device tests (Deltatrac^® vs. QuarkRMR^®) had a power of 0.81 to detect statistical significance.

Results

The subject was a non-smoking 50-year old male, 183 cm tall weighing 82 kg. Body mass index was 24.5kg/m². Resting metabolic rate predicted using the Mifflin St. Jeor equation was 1720 kcal/day (11).

Visual inspection of the three measurements of each test session in Phase 2 revealed a strong tendency for the third measurement to produce a reduction in resting metabolic rate regardless of the device used (Figure 1). In six of ten cases the second measurement was lower than the first. The maximum difference between these pairs was 3% with a mean difference of 1.8%. In contrast, the third measurement was lower than the first nine out of ten times with a maximum difference of 9% and a mean difference of 6.5%, and lower than the second seven of ten times with a maximum difference of 10% and a mean difference of 7.4%. There was no pattern by brand of device. Analysis of variance confirmed that the resting metabolic rate in the third test period was significantly lower than the first two test periods and there was no interaction between test period and device sequence. Since the effect had no relation to the devices or the order of testing, the plan to average the first and third measurements was abandoned and only the first two test runs were analyzed.

Figure 1.

Individual measurements of resting metabolic rate undertaken 24 hours apart in the same subject using two different indirect calorimeters. Test 1, 2, and 3 were conducted without the subject arising between measurements. Sequence 1 was Quark-Deltatrac-Quark, Sequence 2 was Deltatrac-Quark-Deltatrac. Each measurement took 30 minutes and each session was preceded by a 30-minute rest period. Total time for each test session was 120 minutes.

Table 1 shows oxygen consumption, carbon dioxide production, resting metabolic rate, and respiratory quotient data for the Deltatrac^® and QuarkRMR^® devices (Phase 2). Measured oxygen consumption and carbon dioxide production from both devices were normally distributed. No statistically significant differences existed between the devices for any gas exchange parameter. Carbon dioxide production and as a result respiratory quotient was more variable than oxygen consumption and resting metabolic rate. The maximum difference in carbon dioxide production between Deltatrac^® and QuarkRMR^® was 12 mL/min or 5.7% of the Deltatrac^® value. By contrast, the maximum difference in oxygen consumption values was 2.8% and in resting metabolic rate 2.6%.

Table 1.

Gas exchange and metabolic data from Deltatrac^® and QuarkRMR^® indirect calorimetry devices measured sequentially in the same subject (Phase 2).

	Deltatrac®		Quark RMR®			Absolute Difference (value)
	Mean ± SD	Range	Mean ± SD	Range	p-value	Mean ± SD	Range
VO2 (mL/min)a	254 ± 8	240–266	255 ± 9	240–271	0.550	4 ± 2	0–7
VCO2 (mL/min)a	205 ± 7	197–219	203 ± 9	193–225	0.250	4 ± 4	0–12
RMR (kcal/day)a	1749 ± 55	1662–1841	1751 ± 59	1654–1878	0.830	28 ± 17	0–47
RQa	0.81 ± 0.02	0.77–0.83	0.80 ± 0.02	0.77–0.83	0.058	0.01 ± 0.01	0.0–0.03

^aVO₂ oxygen consumption, VCO₂ carbon dioxide production, RMR resting metabolic rate, RQ respiratory quotient

Table 2 consists of the absolute and real differences for the QuarkRMR^® vs. Deltatrac^® inter-device comparison, as a percentage of the Deltatrac^® values in Phase 2 and for the intra-device comparison for the Deltatrac^® in Phase 1 and 3 combined. The mean and range of inter-device differences between Deltatrac^® and QuarkRMR^® for oxygen consumption, carbon dioxide production, resting metabolic rate and respiratory quotient performed repeatedly in the same subject were no larger than the intra-device differences between repeat measurements of the Deltatrac^® in the same subject. The mean difference between 10 pairs of Deltatrac^® measurements was 1.2 ± 2.1% for oxygen consumption and −0.5 ± 4.7% for carbon dioxide production. All intra-device oxygen consumption measurement pairs and seven of ten carbon dioxide production measurement pairs were <5% different. Mean difference between Deltatrac^® and QuarkRMR^® for oxygen consumption was 0.4 ± 2.0% and for carbon dioxide 1.0 ± 2.4%. All inter-device oxygen consumption measurement pairs were <5% different from one another, while inter-device carbon dioxide production pairs were <5% different nine of ten times. The tenth pair was different by 5.7%. This level of performance met the a priori definition for precision.

Table 3 displays the bias results of the QuarkRMR^® compared to the Deltatrac^® measurements. None of the gas exchange parameters were biased, and furthermore the confidence intervals were clustered closely around zero. The results for the Deltatrac^® -QuarkRMR^® pairs were similar to those of the Deltatrac^® - Deltatrac^® pairs.

Discussion

The discontinuation of the Deltatrac,^® the current industry benchmark, has made the validation of a new model indirect calorimeter imperative especially presuming there are only a finite number of the reference instruments still available and fully operational. Comparing against a Deltatrac^® Metabolic Monitor, the QuarkRMR^® was shown in the current study to be precise, unbiased, and accurate.

Two previous studies of the QuarkRMR^® compared to the Deltatrac^® have been published (4,5). Both were cross-sectional studies of multiple subjects and both showed similar results of about a 25 kcal/day mean difference between devices and ±220 kcal/day limit of agreement. The authors of one of these studies concluded that this limit of agreement was unacceptably large, suggesting that the QuarkRMR^® needed refinement (5). There are several important limitations to this study. The subjects were not required to fast before the measurement. The length of time the meal was consumed prior to the measurement and the type and quantity of food consumed would introduce variable thermogenic effects of feeding into the measurement. As measurements could not be conducted simultaneously, the impact of thermogenic effect of feeding may have been changing over time and this would have appeared as differences in measured resting metabolic rate due to the devices. The time allotted for rest was only 15 minutes, but it has been reported that in order to accurately measure resting metabolic rate a period of at least 20 minutes and preferably 30 minutes is necessary (12). Therefore, subjects may have been coming into a resting phase after the start of measurements, and this would have appeared as a difference in measured resting metabolic rate caused by the devices. Finally, it is not known if steady state criteria were met for each of measurements.

The authors of the other previous study of the QuarkRMR^® considered their results to show equivalence between the devices (4). These authors may have based their conclusion on the fact that an intra-device comparison of Deltatrac^® showed a difference (26 ± 93 kcal/day) and limit of agreement (−160 to 213 kcal/day) similar to an inter-device comparison of the Deltatrac^® and QuarkRMR^® (difference −29 ± 110 kcal/day and limit of agreement of −248 to 190 kcal/day). The wide limits of agreement for intra-device comparisons might be explained by a time effect since three indirect calorimetry measurements were taken over 140 minutes including a rest period. In a similar protocol of three measurements over 120 minutes in the current study, there was a sharp drop in resting metabolic rate during the third measurement independent of device. Therefore the wide limits of agreement measured in the Blond study may have been due not to device variation but to variation in the subjects.

In the current study, a different approach from the other studies was taken in that rather than measuring multiple volunteers a single time, one trained individual was tested multiple times with the same device (Deltatrac^® vs. Deltatrac^®) and two different devices (Deltatrac^® vs. QuarkRMR^®). By this method, variation due to the subject was both minimized and defined. The methodology furthermore allowed for a measurement of precision (i.e. the tendency for the same result to be obtained with repeated measurements in the same subject). Under this condition, the QuarkRMR^® was found to be unbiased and precise, producing results that were at most 2.6% different from Deltatrac^® for resting metabolic rate. This variation compares favorably with maximum variation in intra-device measures using the Deltatrac^® (3.2%). Carbon dioxide production and therefore RQ was found to be more variable (maximum 4.3% for QuarkRMR^® vs. Deltatrac^®), but the same was true for comparisons within Deltatrac^® measurements (maximum 9%).

The inter- and intra-device differences in oxygen consumption and carbon dioxide production in the current study were similar to the in vitro differences reported for the Deltatrac^® when it was undergoing validation testing in the 1990s (1,2). For instance, Weissman recorded an oxygen consumption measurement by a Deltatrac^® that was 1.3 ± 1.0% different from a known constant generated in an artificial lung. At a comparable level of oxygen consumption in the current study, two Deltatrac^® measurements conducted consecutively in the same subject and repeated 10 times over 10 days showed a mean difference of 1.3 ± 2.1%. Similarly, QuarkRMR^® measurements conducted in the same subject 10 times over 10 days were 0.4 ± 2.0% different from Deltatrac^® measurements conducted immediately before or after the QuarkRMR^® measurements.

Variation between the devices is best indicated by the absolute differences. The absolute intra-device difference for resting metabolic rate (Deltatrac^®) was 2.0 ± 1.0% and the absolute inter-device difference was 1.6 ± 0.9% (QuarkRMR^® as a percentage of the Deltatrac^® measurement).

Previous attempts at validating other replacement devices are limited and have yielded unfavorable results. In a three-site study, Cooper et al. (3) analyzed the validity and reliability of five instruments, MedGem^®, MedGraphics CPX Ultima^®, Vmax Encore 29^®, TrueOne 2400^® and Korr ReeVue^®, to the Deltatrac II Metabolic Monitor^®. Only the TrueOne 2400^® and Vmax Encore 29^® were valid for measurement of resting metabolic rate, with mean within-subject differences of −6 ± 131 kcal/day and coefficient of variation 5.4% and −26 ± 155 kcal/day and coefficient of variation 8.4% for each device respectively. Neither device proved to be satisfactorily reliable as both had wide limits of agreement of about −400 to 200 kcal/day for resting metabolic rate compared to the Deltatrac^®. The QuarkRMR^® was not available for testing in this study.

Conclusion

Under in vivo conditions in which measurement differences due to biological variation were defined and minimized, the QuarkRMR^® was demonstrated to be unbiased, precise, reproducible, and accurate compared to an indirect calorimeter that is regarded as a criterion method but that is no longer manufactured. The QuarkRMR^® is a valid instrument for measuring gas exchange in spontaneously breathing people. In vitro testing against known constants for oxygen consumption and carbon dioxide production should be conducted to confirm this conclusion.

Table 2.

Table 3. Bias of QuarkRMR^® relative to Deltatrac^® and within multiple measurements with Deltatrac^® (95% confidence intervals of the differences that exclude zero indicate bias).

	QuarkRMR® - Deltatrac®		Deltatrac® 2 – Deltatrac® 1a
	95% confidence interval		95% confidence interval
Parameter	(actual value)	(as percentage of Deltatrac®)	(actual value)	(as percentage of Deltatrac® 1)
VO2 (mL/min)b	−4.6 to 2.6	−1.8 to 1.0	−6.8 to 0.8	−2.7 to 0.3
VCO2 (mL/min)b	−1.7 to 5.7	−0.8 to 2.7	−6.1 to 7.3	−2.9 to 3.8
RMR (kcal/day)b	−27 to 22	−1.5 to 1.2	−43 to 10	−2.5 to 0.6
RQb	−0.0005 to 0.0233	−1.8 to 1.0	−0.15 to 0.04	−1.7 to 4.6

^aDeltatrac^® 2 = the second Deltatrac^® measurement in Phase 1 and 3 when the Deltatrac^® was used twice in the same measurement session. Deltatrac^® 1 was the first measurement of the pair.

^bVO₂ oxygen consumption, VCO₂ carbon dioxide production, RMR resting metabolic rate, RQ respiratory quotient