Iterative outlier removal: A method for identifying outliers in laboratory recalibration studies

Abstract

Background

Extreme values that arise for any reason, including through non-laboratory measurement procedure-related processes (inadequate mixing, evaporation, mislabeling), lead to outliers and inflate errors in recalibration studies. We present an approach termed iterative outlier removal (IOR) for identifying such outliers.

Methods

We previously identified substantial laboratory drift in uric acid measurements in the Atherosclerosis Risk in Communities (ARIC) Study over time. Serum uric acid was originally measured in 1990–92 on a Coulter DACOS instrument using an uricase-based measurement procedure. To recalibrate previous measured concentrations to a newer enzymatic colorimetric measurement procedure, uric acid was re-measured in 200 participants from stored plasma in 2011–13 on a Beckman Olympus 480 autoanalyzer. To conduct IOR, we excluded data points >3 standard deviations (SDs) from the mean difference. We continued this process using the resulting data until no outliers remained.

Results

IOR detected more outliers and yielded greater precision in simulation. The original mean difference (SD) in uric acid was 1.25 (0.62) mg/dL. After four iterations, 9 outliers were excluded, and the mean difference (SD) was 1.23 (0.45) mg/dL. Conducting only one round of outlier removal (standard approach) would have excluded 4 outliers (mean difference [SD] = 1.22 [0.51] mg/dL). Applying the recalibration (derived from Deming regression) from each approach to the original measurements, the prevalence of hyperuricemia (>7 mg/dL) was 28.5% before IOR and 8.5% after IOR.

Conclusion

IOR is a useful method for removal of extreme outliers irrelevant to recalibrating laboratory measurements, and identifies more extraneous outliers than the standard approach.

Introduction

In long-term cohort studies, clinical laboratories and laboratory measurement procedures (i.e., type of assay, instrument, or specimen) may change over time. For instance, different specimen types (plasma versus serum) may be used at different time points, and modern instruments may replace older ones. These changes can result in substantial between-assay variation (laboratory drift). It is crucial to periodically evaluate laboratory drift in longitudinal studies and, if present, to re-calibrate measurements to ensure alignment of values across study visits. This laboratory recalibration is especially important for analyses of trends over time. Proper implementation of a laboratory recalibration study includes parsing out different sources of error. Outliers often occur in such laboratory recalibration studies. Extreme values that arise through a process unrelated to the laboratory measurement procedure, such as inadequate mixing, evaporation, degradation, mislabeling, or data entry errors are not informative about the recalibration of the vast majority of samples where such errors did not occur. In fact, these outliers can decrease precision and increase the error in the derived recalibration regression. Therefore, distinguishing between error related and unrelated to the laboratory measurement procedure is critical but challenging.

The general principles of how one might handle outliers in clinical chemistry applications have been previously outlined.¹ More specifically, various approaches have been described to identify and minimize the effect of outliers when comparing measurements in laboratory recalibration studies.^2–9 These approaches must be balanced with the risk of omitting too many data points and artificially improving agreement. In general, it is good practice to evaluate and report the percentage of outliers identified and to evaluate the laboratory measurement procedure error after exclusion of these outliers. We present here an Iterative Outlier Removal (IOR) method, a straightforward and efficient approach to identifying outliers that are likely irrelevant to the recalibration process. We use data from a laboratory recalibration study on uric acid in a large prospective cohort, the Atherosclerosis Risk in Communities (ARIC) Study, to demonstrate the utility of the IOR approach; and conduct a simulation to compare the effect of IOR to that of a standard outlier removal approach.

Methods

Study population

The ARIC Study is an ongoing community-based cohort of 15,792 adults from four communities in the United States.¹⁰ There have been five main clinic examinations. The first examination (visit 1) took place from 1987–1989 with three follow-up examinations occurring approximately three years apart during 1990 to 1992 (visit 2), 1993 to 1995 (visit 3), and 1996 to 1998 (visit 4). A fifth examination was completed in 2011–13. An institutional review board at each site approved all procedures, and all study participants provided written informed consent.

We previously undertook a laboratory recalibration study to align measurements taken over the five ARIC visits that were conducted in two different laboratories at different times up to 25 years apart, using different assays, instruments, and specimen types.¹¹ A recalibration subsample consisting of blood samples from 200 ARIC participants was selected for re-assay of blood analytes in stored specimens. We present here the results for the recalibration of uric acid measurements at visit 2 to demonstrate the IOR method for outlier identification and exclusion.

Laboratory measurement procedures

Uric acid was originally measured in serum at visit 2 (1990–92) at the University of Minnesota using the Coulter DACOS system using an uricase measurement procedure (inter-assay CV=2.7%).¹² The recalibration study was performed on the 200 samples from each ARIC visit in stored plasma in 2011–13 at Baylor College of Medicine using the Beckman Olympus AU 480 using an enzymatic colorimetric measurement procedure (CV=1.8%). For this paper, we use only visit 2 measurements.

Iterative Outlier Removal (IOR) method

To identify outliers unrelated to laboratory measurement procedure error and likely not relevant to the recalibration process, we undertook the following steps to compare the uric acid values measured in 1990–92 to those measured in 2011–13 in the 200 participants with both available values:

1. We calculated the mean difference, where the difference for each participant was calculated as the original value minus the re-measured value. We identified and excluded data points for which the difference was >3 standard deviations (SDs) away from the mean difference.

2. We then calculated the mean difference and corresponding SD after excluding the previously identified outlying data points. We identified and excluded data points for which the difference was > 3 SDs away from this new mean difference. We repeated step #2 and continued this iterative process until no outliers remained.

The IOR process was considered complete when all differences lie within the 3 SDs of the mean difference. When non-constant bias is suspected since the slope of the regression of new values on old values is very different from 1.0, this method can be applied to the residual from the regression rather than differences.

Comparison with standard approach using ARIC data and simulations

For comparison, we used a standard approach of one round of identifying data points outside of the 3 SDs from the mean difference. We compared the mean values, mean difference and SD, recalibration equations, and prevalence of hyperuricemia (uric acid >7.0 mg/dL) that resulted from the standard and IOR approaches.

As a second method of comparing the implications of using a standard approach of conducting one round of outlier removal to IOR, we conducted 500 rounds of simulations. To do this, we created a data set of 1,000 observations randomly selected from a normal distribution with mean 100 and SD=10 as the unobserved true values. To simulate the original values, we added measurement error with a normal distribution with mean zero and SD=10. To simulate the reference value measured later, we added two sources of error: (1) measurement error with a normal distribution with mean zero and SD=10 that is independent of the original measurement and (2) outlier error with a normal distribution with mean of +/−60 and SD=10 with this latter part only included randomly with a fixed frequency (Bernoulli distribution with probability of either 1%, 5%, or 10% simulating a higher fraction of outliers) and 50/50 distribution of positive and negative sign. The difference between the reference and true measurements reflects the total amount of measurement error. We also conducted supplemental analyses that specified fixed error rates of 0%, 1%, 5%, 10%, 20%, 30%, 40%, and 50%; and an asymmetric distribution of the outliers (75% and 100% of the outliers on the positive side of the sample).

For each of the simulation scenarios we removed outliers using two approaches: a standard approach of one round of removing outliers >3 SDs from the mean difference, and the IOR approach. We then calculated the mean and SD of the difference of the two measurements, the slope of the difference versus the mean and its standard error, and the intercept of the difference versus the mean and its standard error; and then calculated the mean and SD, as well as the median and 5^th and 95^th percentiles of each of these results. For each of the outlier removal approaches, we also calculated the proportion of observations that were identified as outliers and removed.

RESULTS

We present descriptive statistics of the original and reference uric acid measurements (Table 1). The scatterplot of the uric acid reference measurement against the original measurement, including the line of equality (Y=X), Pearson correlation coefficient, and regression coefficients and SDs are shown in Figure 1. To illustrate the IOR method, we present a Bland-Altman plot that indicates the outliers that were identified and excluded after each iteration of IOR (Figure 2). We conducted four iterations of IOR before we obtained a sample with no values greater than 3 SDs away from the mean difference. We identified 9 outliers (4.5% of data points) after four iterations of IOR for uric acid.

Table 1.

Descriptive statistics of uric acid before outlier removal, after one round of outlier removal, and after iterative outlier removal

	Before IOR (N=200)	After 1 round of outlier removal (N=196)	After IOR (N=191)
	Mean (SD) (mg/dL) or %	Mean (SD) (mg/dL) or %	Mean (SD) (mg/dL) or %
Original	6.41 (1.44)	6.35 (1.39)	6.36 (1.38)
Reference	5.17 (1.30)	5.13 (1.23)	5.12 (1.23)
Difference (Original-Reference)	1.25 (0.62)	1.22 (0.51)	1.23 (0.45)
Mean of Original and Reference	5.79 (1.34)	5.74 (1.28)	5.74 (1.29)
Hyperuricemia, %*	28.5%	7.5%	8.5%

Abbreviations: IOR, iterative outlier removal

^*Hyperuricemia defined as uric acid >7.0 mg/dL

Figure 1. Scatterplots of reference versus original uric acid measurements, before and after iterative outlier removal.

Pearson correlation coefficients for the reference versus original uric acid measurements are presented in each of the scatterplots. Beta coefficients and corresponding standard errors for the regression line are also reported.

Figure 2. Bland-Altman plot of uric acid with all outliers identified during iterative outlier removal.

The horizontal dashed line indicates a difference of zero. The solid black lines indicate the mean difference and +/− 2 standard deviations from the mean difference (calculated after exclusion of outliers). The dashed line indicates the regression line (after exclusion of outliers). We present outliers identified during each iteration of iterative outlier removal (IOR).

IOR identified 4.5% of data points as outliers. We re-calculated descriptive statistics and regenerated a scatterplot after exclusion of these outliers. The mean difference (original minus re-measured measurements) was 1.25 mg/dL before IOR and 1.23 mg/dL after IOR (Table 1). The SD of the mean difference was notably larger before versus after IOR (0.62 versus 0.45, respectively (Table 1). After IOR, the Pearson correlation coefficient was higher (0.947 versus 0.902) and the standard error of the regression line coefficient was smaller, as can be seen in the comparison of the scatterplots (Figure 1). Additionally, the standard error of the regression of the mean difference versus the mean of the uric acid measurements became smaller after each iteration of IOR (Table 2).

Table 2.

Beta coefficients and standard errors of the regression lines of the difference versus mean uric acid measurements after each iteration of IOR

	N remaining	Beta (SE)
Before IOR	200	0.10 (0.032)
After 1st iteration	196	0.13 (0.027)
After 2nd iteration	194	0.12 (0.026)
After 3rd iteration	192	0.12 (0.025)
After 4th iteration	191	0.12 (0.024)

Abbreviations: IOR, iterative outlier removal; SE, standard error

The beta coefficients and SEs were obtained from the regression line of the difference between uric acid measurements (original minus reference) versus the mean of uric acid measurements

We then used Deming regression to calibrate the original measurement to the reference measurement, using the data set that excluded all outliers identified by IOR. The resulting recalibration equation was: calibrated uric acid = −0.47 + 0.88*original uric acid.

If we had only conducted one round of outlier removal, we would have identified four outliers, instead of nine. After exclusion of these four outliers, the resulting recalibration equation from Deming regression would have been: −0.43 + 0.87*original uric acid. After applying this recalibration equation to the original uric acid measurements in the 200 participants, the prevalence of hyperuricemia was slightly lower than when using IOR (Table 1).

Results of our simulation study further support the use of IOR as a useful method for identifying and removing outliers. As the proportion of measurements with both measurement error and additional error from other processes increased from 1% to 10%, the inflation in the SD of the difference between measurements increased (median SDs were 15, 20, and 24 for error rates of 1, 5, and 10%, respectively) (Table 3). After conducting only one round of outlier removal, the issue of inflated SDs remained, but was not quite as severe. Further proceeding with multiple iterations of outlier removal using the IOR approach resulted in SDs that were no longer inflated (median SDs were 14.0, 14.3, and 14.8) (Table 3). IOR detected more outliers than conducting just one round of outlier removal (median proportion of observations identified as outliers were 1.10 vs 0.9%, 4.4 vs 2.6%, and 8.3 vs 2.5%, for error rates of 1, 5, and 10%, respectively) (Table 3). IOR even performed well in the setting of asymmetric distribution of the outliers, as evidenced by SDs that were no longer inflated and a greater proportion of outliers observed compared to the standard approach (Supplemental Table 1). However, when the rate of outliers was very high (20% or greater) the IOR method no longer worked, presumably since the outliers now changed the whole distribution; SDs remained inflated and the proportion of outliers detected was low (median proportion of observations identified as outliers were 1.0%, 0.1%, 0.0%, and 0.0% for 20%, 30%, 40%, and 50% error rates, respectively) (Supplemental Table 2).

Table 3.

Simulation to compare original and reference measurements before outlier removal, after one round of outlier removal, and after iterative outlier removal

Bernoulli success probability:		p=0.01	p=0.05	p=0.10
Before outlier removal (N=1,000)
Mean of Difference	Mean (SD)	0.01 (0.49)	0.04 (0.62)	0.06 (0.74)
	Median (5th–95th)*	0.03 (−0.78, 0.80)	0.09 (−0.96, 1.04)	0.10 (−1.31, 1.21)
SD of Difference	Mean (SD)	15.4 (0.5)	19.6 (0.8)	23.9 (0.9)
	Median (5th–95th)*	15.4 (14.6, 16.2)	19.6 (18.3, 20.9)	23.9 (22.4, 25.4)
Slope of difference vs. mean	Mean (SD)	0.12 (0.05)	0.47 (0.07)	0.76 (0.06)
	Median (5th–95th)*	0.12 (0.03, 0.21)	0.47 (0.36, 0.58)	0.77 (0.66, 0.86)
Standard error of Slope of   difference vs. mean	Mean (SD)	0.04 (0.00)	0.04 (0.00)	0.04 (0.00)
	Median (5th–95th)*	0.04 (0.04, 0.04)	0.04 (0.04, 0.04)	0.04 (0.04, 0.04)
Intercept of difference vs. mean	Mean (SD)	0.01 (0.48)	0.04 (0.57)	0.04 (0.64)
	Median (5th–95th)*	0.03 (−0.77, 0.78)	0.06 (−0.98, 0.93)	0.07 (−1.05, 1.10)
Standard error of Intercept of   difference vs. mean	Mean (SD)	0.48 (0.01)	0.58 (0.02)	0.65 (0.02)
	Median (5th–95th)*	0.48 (0.46, 0.51)	0.58 (0.55, 0.61)	0.65 (0.63, 0.68)
After one round of outlier removal
Mean of Difference	Mean (SD)	0.02 (0.45)	0.03 (0.55)	0.07 (0.72)
	Median (5th–95th)*	0.03 (−0.70, 0.73)	0.03 (−0.83, 0.90)	0.06 (−1.10, 1.23)
SD of Difference	Mean (SD)	14.1 (0.3)	15.8 (0.6)	20.3 (1.2)
	Median (5th–95th)*	14.1 (13.5, 14.7)	15.7 (14.8, 16.9)	20.3 (18.4, 22.3)
% Observations identified as   outliers and removed	Mean (SD)	0.91 (0.25)	2.62 (0.31)	2.49 (0.35)
	Median (5th–95th)*	0.90 (0.50, 1.35)	2.60 (2.10, 3.10)	2.50 (1.90, 3.05)
Slope of difference vs. mean	Mean (SD)	0.02 (0.04)	0.19 (0.06)	0.55 (0.08)
	Median (5th–95th)*	0.02 (−0.05, 0.08)	0.19 (0.09, 0.29)	0.56 (0.42, 0.69)
Standard error of Slope of   difference vs. mean	Mean (SD)	0.04 (0.00)	0.04 (0.00)	0.04 (0.00)
	Median (5th–95th)*	0.04 (0.03, 0.04)	0.04 (0.04, 0.04)	0.04 (0.04, 0.04)
Intercept of difference vs. mean	Mean (SD)	0.02 (0.45)	0.02 (0.53)	0.06 (0.65)
	Median (5th–95th)*	0.04 (−0.70, 0.72)	0.04 (−0.82, 0.87)	0.07 (−1.02, 1.15)
Standard error of Intercept of   difference vs. mean	Mean (SD)	0.45 (0.01)	0.50 (0.02)	0.59 (0.02)
	Median (5th–95th)*	0.45 (0.43, 0.47)	0.50 (0.47, 0.53)	0.59 (0.56, 0.63)
After IOR
Mean of Difference	Mean (SD)	0.02 (0.45)	0.02 (0.50)	0.01 (0.53)
	Median (5th–95th)*	0.05 (−0.67, 0.77)	0.02 (−0.77, 0.83)	0.03 (−0.88, 0.85)
SD of Difference	Mean (SD)	14.0 (0.3)	14.3 (0.4)	14.8 (0.6)
	Median (5th–95th)*	14.0 (13.4, 14.5)	14.3 (13.6, 15.0)	14.8 (13.9, 15.8)
% Observations identified as   outliers and removed	Mean (SD)	1.16 (0.35)	4.45 (0.72)	8.30 (0.92)
	Median (5th–95th)*	1.10 (0.60, 1.70)	4.40 (3.20, 5.60)	8.30 (6.80, 9.85)
Slope of difference vs. mean	Mean (SD)	0.01 (0.04)	0.05 (0.04)	0.12 (0.06)
	Median (5th–95th)*	0.01 (−0.05, 0.07)	0.05 (−0.02, 0.12)	0.12 (0.03, 0.22)
Standard error of Slope of   difference vs. mean	Mean (SD)	0.04 (0.00)	0.04 (0.00)	0.04 (0.00)
	Median (5th–95th)*	0.04 (0.03, 0.04)	0.04 (0.03, 0.04)	0.04 (0.04, 0.04)
Intercept of difference vs. mean	Mean (SD)	0.02 (0.45)	0.02 (0.49)	0.01 (0.52)
	Median (5th–95th)*	0.05 (−0.68, 0.76)	0.02 (−0.76, 0.81)	0.04 (−0.86, 0.83)
Standard error of Intercept of   difference vs. mean	Mean (SD)	0.44 (0.01)	0.46 (0.01)	0.49 (0.02)
	Median (5th–95th)*	0.45 (0.43, 0.46)	0.46 (0.44, 0.48)	0.49 (0.46, 0.51)

We conducted 500 simulations, specifying a bimodal normal distribution with a mean of +/−60 and a standard deviation of 10.

^*We present the median and corresponding 5^th and 95^th percentiles, or the empirical 90% confidence interval.

In the case of asymmetric distribution of the outliers, IOR performed better than the standard approach.

Discussion

We recommend the use of an IOR approach to identify outliers that are most likely due to processes that are different from the laboratory measurement procedure, such as sample evaporation and/or degradation or specimen mislabeling. It is important to clarify that exclusion of outliers is not conducted to artificially improve agreement of measurements. Laboratory recalibration is intended to correct for laboratory measurement procedure-related processes such as assays being conducted using different instruments, in different laboratories, or different specimen types. Therefore, it is useful to remove outliers arising from processes that are either different from laboratory measurement procedures or may be the result of an issue with the measurement procedure itself, such as interference that is larger than usual measurement error and hence produces outliers.

A classical measurement error model assumes that the error is normally distributed with a given variance. However, we have assumed that the error is a mixture of a normal distribution with a small variance and a normal distribution with a large variance (errors related and unrelated to laboratory-measurement procedure variability, respectively). IOR can help distinguish data points resulting from these two types of errors, and intends to exclude data points coming from the distribution with the larger variance. This formulation also applies to the calculation of the difference of the original and reference measurements.

The purpose of any approach to outlier removal is to minimize the influence of error due to factors other than those we are trying to account for during the recalibration process. Including measurements due to extraneous error processes could introduce unwanted noise into a recalibration equation. Indeed, we noted that after each iteration of IOR, the standard error of the regression line coefficient decreased. Previous methods have typically relied on one or two rounds of outlier removal, which attempts to remove extreme outliers.^3–6 However, these methods have not assessed the distribution of remaining data points after these outliers are removed, and the potential for any residual unrelated error. Our iterative approach proposes to remove any additional remaining extreme values that are due to these extraneous error processes. Compared to a standard approach of conducting one round of outlier removal, IOR enables us to identify a greater proportion of these outliers that are irrelevant to recalibration. We are then able to conduct Deming regression or implement another appropriate recalibration approach using only those data points that are representative of the expected laboratory measurement procedure variability, resulting in a more accurate recalibration equation. However, the use of IOR is limited in the setting of a high error rate (≥20%), in which it did not perform any better than the standard approach. Furthermore, as has previously been noted,¹ if outliers are too close to the 3 SD threshold, they will not be identified by the IOR approach, since they will influence the estimate of the SD. Whereas differences in results between IOR and conducting one round of outlier removal may seem subtle, we propose that IOR results in a more accurate recalibration of measurements in many situations.

It is crucial to use outlier removal methods that are transparent and easily replicated, and can be easily communicated and understood by the scientific community. IOR fulfills these criteria and can be conducted using standard statistical software. We have demonstrated this accessible approach to identify data points that are extraneous to the recalibration process, which may be useful for clinical chemists, epidemiologists, lab scientists, and other investigators who have data that require recalibration.