Reliability of Plasma Carotenoid Biomarkers and Its Relation to Study Power

Abstract

Background

The reliability of biomarkers profoundly impacts validity of their use in epidemiology and can have serious implications for study power and the ability to find true associations. We assessed reliability of plasma carotenoid levels over time and how it could influence study power through sample size and effect-size.

Methods

Plasma carotenoid levels were measured in a cohort study of 1323 women participating in the control arm of the Women's Healthy Eating and Living Study. We compared mean plasma levels at baseline, year 1, and year 4 of the study for alpha-carotene, beta-carotene, lycopene, lutein, and beta-cryptoxanthin. Reliability of these levels over time was assessed by Spearman correlations and intraclass correlation.

Results

We found limited variation in mean levels between any 2 time points. Variation did not exceed 8% for lycopene, lutein, and beta-cryptoxanthin, 15% for alpha-carotene, and 18% for beta-carotene. Spearman correlations for individual carotenoids over time varied between 0.50 and 0.80, with lycopene having the lowest correlation. Intraclass correlations ranged from 0.47 to 0.66 for carotenoids.

Conclusion

Intraclass correlations for plasma carotenoids over a period of several years are acceptable for epidemiologic studies. However, such variation is enough to decrease statistical power and increase the sample size needed to detect a given effect.

Validity and reliability are 2 critical features of any measurement tool. Validity is defined as the ability to measure the true exposure of interest; while reliability is an index of the consistency with which the instrument measures the exposure.¹ Even a valid biomarker will be of limited value if it fails to reliably reflect the exposure of interest. If a measurement method does not have high reliability, relative risk estimates maybe biased and the statistical power of the study maybe affected.^1-6

The issue of biomarker reliability is especially important for cohort studies using prospectively-collected data, because cost or logistical constraints may limit the frequency or extent to which samples are collected.⁷ If a biomarker is highly reliable, it can serve to reflect the exposure of interest in a population over time.

Carotenoids are pigments provided mainly from fruit and vegetables in the diet.⁸ Their plasma levels are biomarkers of fruit and vegetable intake,^9-12 and can directly exhibit a range of biologic activities.^13-16 In this analysis, we assessed reliability of plasma carotenoid levels over time in a cohort of breast cancer survivors participating in the Women's Healthy Eating and Living Study.¹⁷ Dietary and lifestyle data and blood samples analyzed for carotenoid levels were sequentially collected over time, which offers a unique opportunity to determine the reliability of carotenoids among carefully monitored women in a longitudinal setting. We assessed the carotenoid levels among women participating in the control arm of the intervention, in which dietary intakes of vegetables and fruit were reported to be stable over time.^9,18

METHODS

Study Population

The Women's Healthy Eating and Living Study is a multisite randomized controlled trial of the effectiveness of encouraging a diet high in fruits and vegetables and low in fat intake in reducing additional breast cancer events and early death among women with early-stage invasive breast cancer. A total of 3088 breast cancer survivors with a mean age of 53 years from 7 sites in California, Arizona, Texas, and Oregon were recruited between 1995 and 2000 and randomly allocated to either an intervention or comparison group. The intervention group was encouraged to adopt a plant-based, low-fat dietary pattern through a telephone counseling protocol.¹⁷ Comparison group participants were encouraged to continue to maintain healthy eating habits; they did not receive individualized counseling.¹⁹

Dietary intake, measures of height and weight, and changes in personal habits and health symptoms were collected by telephone and during clinic visits at baseline, 12 months, 24 or 36 months (50% at each of those time points), 48 months, and 72 months.¹⁷ In this analysis, we used carotenoid plasma levels from 3 time points with complete data collection: baseline, 1 year, and 4 years follow-up for the comparison group. To assess the reliability of plasma carotenoid levels, we included all participants from the comparison group, which on average maintained stable carotenoid levels during the 4 years from randomization.⁹ Data from 1323 women with plasma carotenoid data were analyzed. We did not include the intervention arm of the study because the intervention produced large changes in plasma carotenoid levels over time. The institutional review boards for all participating institutions approved the procedures for this study, and written informed consent was obtained from all study participants.

Laboratory Assay of Plasma Carotenoids

Fasting blood samples were collected from all participants during the clinic visits and immediately placed on ice, protected from light, and separated within 1 hour after collection, using centrifugation at 2300g at 4°C for 10 minutes. Aliquots of plasma and serum were stored at −70°C in cryogenic tubes until analysis. Plasma aliquots were analyzed for concentrations of 5 plasma carotenoids (alpha-carotene, beta-carotene, lutein plus zeaxanthin, lycopene, and beta-cryptoxanthin), which comprise more than 90% of the carotenoids in human plasma. Plasma carotenoids were separated and quantified using high-performance liquid chromatography methodology. This analysis was conducted with a Varian Star 9010, 9050 system with variable wavelength ultraviolet/visible light detector (Varian Analytical Instruments, Walnut Creek, CA) with wavelength set at 450 nm. The mobile phase is acetonitrile/methanol/methylene chloride (70:10:30, vol/vol/vol), with triethylamine (0.13 mL/L acetonitrile) and ammonium acetate (0.1 g/L methanol) modifiers used to enhance recovery. The column is a Supelco (Bellefonte, PA) Supelcosil LC-18 (25 cm × 4.6 mm × 5 μm). This analytic method measures 90% of the total plasma carotenoids and permits quantification of the predominant carotenoids. Zeaxanthin and lutein elute together with this methodology, so values presented as lutein represent lutein plus zeaxanthin. Accuracy was assessed by periodic analysis of National Institute of Standards and Technology Standard Reference Material SRM 986. Fat-soluble vitamins, and a pooled plasma reference sample was analyzed concurrently with batches of study samples to monitor analytical precision, with day-to-day coefficient of variation ≤7%. Additionally, the laboratory participated in the National Institute of Standards and Technology Round-Robin Quality Assurance program for fat-soluble micronutrients.

For each sample designated for carotenoid analysis, determinations of total plasma cholesterol concentrations were performed with the Kodak Ektachem Analyzer system (Johnson & Johnson Clinical Diagnostics, Rochester, NY).²⁰ The evaluation of carotenoid data accounts for plasma cholesterol levels, because cholesterol-carrying lipoproteins are important determinants of the circulating pool of these compounds.²¹

Statistical Analysis

Descriptive characteristics (mean, SD, median) of each of the individual plasma carotenoid markers and total plasma carotenoids were calculated. Multiple measures of reliability were used to obtain a comprehensive assessment of consistency of plasma carotenoid markers over time. Spearman correlations between each pair of baseline, 1-year, and 4-year plasma values were computed. In addition, intraclass correlations (ICC) and within-subject coefficient of variation²² were estimated using mixed models. The models used are described below.

Classic reliability studies assume a steady-state measure in the subjects. This assumption is untenable when studying factors such as diet or quality of life, which may vary over time for the participant. We adopt the approach of Laenen et al²³ to estimate the “generalized reliability” of the plasma carotenoid marker. The mixed modeling paradigm permits a fairly general model, allowing for changes in the participant's plasma level over time by adjustment for covariates (eg, time) and by incorporating general variance-covariance error structures.

The model and underlying modeling assumptions are illustrated below. Let Y_ij be the plasma marker value for subject i at time j (where j = 0, 1, 4 representing baseline, 1- and 4-year measures). The model was defined as Y_ij = α_i + (covariates) + error_ij, where α_i represents a subject-specific random-effect term assumed to be distributed normally with mean μ and variance τ². The errors (error_ij) are assumed independent of α_i, and are also distributed normally with mean 0 and variance σ². Covariates included in the models as fixed effects were age at randomization, year of study (baseline, year 1 or year 4), plasma cholesterol, body mass index (BMI), clinical site, alcohol use (tertiles), season of blood draw, smoking status (current vs. not current) at each of the time points. We did not include reported dietary intake of foods contributing to plasma carotenoids, since they represent the same exposure of plasma carotenoids we were trying to assess for this study.

To allow flexible modeling, we fit several models with different covariance structures for the vector (error_i0, error_i1, error_i4), including diagonal, compound symmetry, autoregressive, and unstructured. A diagonal covariance structure assumes that errors in plasma measurements are uncorrelated over time [ie, that corr(error_ij,error_ik) = 0 for all k different from j]. This is a stringent assumption, and it is possible that the plasma measurement process has systematic errors that repeat for every assay. By using different covariance matrices for the vector of errors, we can incorporate the possibility of such systematic errors in our models. In particular, a compound symmetry model would posit that corr(error_ij,error_ik) = ρ for all k different from j, suggesting that error correlations are constant (and possibly nonzero) over time; in contrast, the autoregressive assumption states that corr(error_ij,error_ik) = ρ^(j–k) for all k different from j, which implies that error correlations are higher for measurements taken closer together in time compared with those obtained farther apart. Finally, the unstructured covariance structure makes no assumptions about the dependence of error correlations on time, yielding in our model 3 additional estimated parameters: ρ₁₂, ρ₁₃, ρ₂₃, where ρ_jk = corr(error_ij,error_ik). Model fit was assessed by plotting residuals against fitted values, and also by graphing QQplots of the residuals to check that Gaussian error assumptions were satisfied.

Using these models, the intraclass correlation is defined as ICC = τ²/(τ² + σ²). The impact of ignoring nonzero error correlations can be assessed as follows. For instance, if the true model had a compound symmetric error covariance matrix, then assuming a diagonal error covariance matrix would estimate the intraclass correlation as ICC(diagonal) = (τ² + ρ × σ²)/(τ² + σ²), where ρ represents the correlation between error terms at different time points. Thus, ICC (diagonal) = ICC + (ρ × σ²)/(τ² + σ²), would be an overestimate of the true ICC, if ρ > 0 (ie, if errors were positively correlated). More general models with autoregressive and unstructured error covariance matrices provide additional sensitivity analysis regarding assumptions about the error process over time.

The within-subject coefficient of variation is defined as σ/μ.²² The ICC is scale-invariant and measures the proportion of true-signal variability in the assessment method. However, it is population dependent.²² The within-subject coefficient of variation, although not scale invariant, provides an additional measure of consistency and focuses on within-subject variability.

RESULTS

At baseline, the median age of the Women's Healthy Eating and Living Study participants included in this report was 53 years, with a range of 27−74 years. The median BMI was 26 (range 15−56) kg/m² and 86% were non-Hispanic white.

As shown in Table 1, mean levels for individual carotenoids were relatively stable over time. With the exception of alpha-carotene and beta-carotene between baseline and year 4, and between year 1 and year 4, there was negligible change in mean plasma carotenoid level; relative mean changes ranged from 0% to 8% (Table 2). The least change in mean plasma levels for any of the carotenoids was between baseline and year 1.

TABLE 1.

Descriptive Statistics of the Plasma Measures Over Time (umol/L)

Carotenoid	Baseline (n = 1205) Mean (SE)	Year 1 (n = 1183) Mean (SE)	Year 4 (n = 1039) Mean (SE)
Alpha-carotene	0.208 (0.006)	0.212 (0.006)	0.171 (0.005)
Beta-carotene	0.936 (0.030)	0.901 (0.027)	0.800 (0.024)
Lutein	0.382 (0.006)	0.390 (0.006)	0.371 (0.006)
Lycopene	0.656 (0.01)	0.648 (0.01)	0.682 (0.009)
Beta-cryptoxanthin	0.181 (0.005)	0.181 (0.004)	0.196 (0.005)
Total carotenoids	2.364 (0.042)	2.332 (0.040)	2.220 (0.036)

TABLE 2.

Relative Change in Mean Plasma Carotenoid Levels and Spearman Correlations at Different Time Points

	Baseline to Year 1		Baseline to Year 4		Year 1 to Year 4
Carotenoid	% Change in Mean	Spearman Correlation	% Change in Mean	Spearman Correlation	% Change in Mean	Spearman Correlation
Alpha-carotene	2	0.77	−18	0.62	−19	0.67
Beta-carotene	−4	0.80	−15	0.64	−11	0.71
Lutein	2	0.80	−3	0.68	−5	0.66
Lycopene	−1	0.63	4	0.51	5	0.50
Beta-cryptoxanthin	0	0.75	8	0.57	8	0.62
Total carotenoids	−1	0.80	−6	0.64	−5	0.68

Similarly, the correlation of individual plasma carotenoid levels between the different time points was relatively high (Table 2). The highest correlations were between baseline and year 1 (0.63−0.80). The strength of this correlation weakened over time, and the correlations were consistently lower between baseline and year 4 (0.51−0.68) or year 1 and year 4 (0.50−0.71). Lycopene plasma levels had the weakest correlations over time.

We assessed intraindividual variability over time by estimating the ICC using mixed models adjusting for other variables: BMI, plasma cholesterol level, clinical site, alcohol use, season of blood collection, smoking status, and year of blood collection (baseline, year 1, or year 4) (Table 3). These covariates were chosen because they could influence individual plasma carotenoid levels. Including time in the models allowed us to relax the steady-state assumption of classic reliability studies.²³ The ICC were moderate, ranging from 0.47 to 0.66 in the model that assumed uncorrelated errors over time (ie, a diagonal error covariance matrix). When we allowed the error covariance matrix to be unstructured, the ICC ranged from 0.45 to 0.64, suggesting that errors were not strongly correlated. Thus, the independent errors assumption for the plasma marker appears to be reasonable. The ICC for the autoregressive model were similar and hence are not presented. The within-subject coefficient of variation was relatively high (ranging from 28% to 50%), indicating large within-subject variability (Table 3). We repeated the above analysis limited to baseline and 1-year data. As expected for the shorter time period, the ICC was higher, ranging from 0.56 to 0.75; the within-subject coefficient of variation was slightly lower, ranging from 25% to 45%.

TABLE 3.

Intraclass Correlations (ICC) and Within-Subject Coefficient of Variation (WCV) Between Baseline, Year 1, and Year 4

	Between Baseline, Year 1, and Year 4		Between Baseline and Year 1
	ICC	WCV	ICC	WCV
Alpha-carotene	0.63	0.49	0.71	0.43
Beta-carotene	0.66	0.51	0.75	0.42
Lutein	0.65	0.30	0.72	0.28
Lycopene	0.47	0.44	0.56	0.42
Beta-cryptoxanthin	0.59	0.49	0.68	0.45
Total carotenoids	0.66	0.28	0.74	0.25

Implications for Power and Study Design

We examined the implications of these measures for study design. In particular, we considered a study examining the association between a continuous covariate and a binary outcome (ie, a logistic or survival model). We used “effect-size” to quantify the strength of the relationship between exposure and outcome.²⁴ Let n be the sample size required to achieve 80% power to detect a certain effect-size (a log-odds or log-hazard ratio per unit increase in the covariate), assuming the covariate can be measured perfectly reliably (ie, with an intraclass correlation of 1). If the intraclass correlation is actually less than 1, then the required sample size to achieve the same power to detect the same effect-size is proportional to n/ICC.

The figures illustrate some of the effects of poor reliability in relation to the power of the study. Figure 1 provides curves of power versus effect-size and Figure 2 shows power versus sample size, with intraclass correlation values ranging from 0.45 to 0.65 (representing the approximate range of ICC observed). In Figure 1, the top curve depicts a perfect instrument (ICC = 1), that has 90% power to detect an effect-size of 1 with a sample size of 42. As the intraclass correlation decreases, the power to detect an effect-size of 1 decreases, ranging from 74% (ICC = 0.65) to 58% (ICC = 0.45). Conversely, the minimum effect-size that can be detected with 90% power is 1.25 for an instrument with ICC = 0.65, and 1.5 for an instrument with ICC = 0.45. In Figure 2, the top curve represents a perfect instrument (ICC = 1), which has 90% power to detect an effect-size of 1 with a sample size of 42. As the intraclass correlation decreases, the sample size needed to detect an effect-size of 1 with 90% power increases to 64 (ICC = 0.65) or 96 (ICC = 0.45).

FIGURE 1.

Power versus effect-size curves for different values of intraclass correlations from women participating in the Women's Healthy Eating and Living study (sample = 42).

FIGURE 2.

Power versus sample-size curves for different values of intraclass correlations from women participating in the Women's Healthy Eating and Living study (effect size = 1).

DISCUSSION

We observed good reliability of plasma carotenoid levels over time, based on 3 time points 1 to 4 years apart. These samples were from a relatively large cohort in which sample collection, storage, and laboratory analyses were uniform and standardized.

Although there is a slight decline in levels of some carotenoids and an increase in others, this was not more than 18% change even after 4 years. Such variability is acceptable, given the relatively long duration of follow-up. Previous studies have had similar findings.^25-28 Our within-subject coefficient of variation were similar to coefficients of variation reported by Olmedilla et al, who reported coefficients of variation ranging from 29% to 47% for the individual carotenoid levels (although they had a sample of only 18 individuals).²⁸ The coefficients of variation measured within a few days did not exceed 10%. However, a few days do not capture possible long-term intraindividual variability that is influenced by behavioral and biologic factors. The within-subject coefficient of variation reflects a person's biologic variability and intake without accounting for all other factors. As with the difference in mean and Spearman correlation, the within-subject coefficients of variation emphasizes 1 component of variability. The ICC is more relevant to epidemiologic studies of diet-disease associations than within-subject CV because it takes into account both between and within-subject variability.

Other previous studies have tested the reliability of carotenoids only over a period of few weeks.^25,27 However, long-term reliability assessment is more relevant to epidemiologic studies and so reliability data over a period of few years are needed. In 1 study with longer follow-up, the percent difference in individual carotenoids measured 15 years apart did not exceed 26% for any carotenoid.²⁶

Alpha-carotene and beta-carotene had higher variability in mean plasma levels over time than other cartenoids; this might be explained by a stronger association with consumption of specific fruits and vegetables that can lead to intraindividual fluctuation of plasma levels over time. For example, in our study, alpha-carotene was the carotenoid with the highest mean change over time, owing to an increase in carrot intake.⁹

Plasma alpha-carotene and beta-carotene are strongly correlated with each other but not necessarily with other carotenoids.¹⁰ The within-subject coefficients of variation for total carotenoids (25%−28%) was lower than for individual carotenoids, suggesting that there is more individual consistency over time in overall amounts of fruits and vegetables, whereas consumption of individual fruits and vegetables may vary. This information could be useful for studies focusing on individual nutrients in diet, emphasizing the need to assess overall reliability. The intraclass correlation overcomes the limitations of means, within-subject coefficients of variation, and correlation coefficients as measures of reliability. Furthermore, mixed models can be used to adjust the intraclass correlation for age, BMI, alcohol use, and other covariates that change over time. This can provide a general framework in which to assess reliability.²³ The intraclass correlation values for plasma carotenoids from our study are within the range of reliability for other biomarkers measured several years apart, such as lipid biomarkers and markers of oxidative damage.⁷ The lower intraclass correlation with longer time periods of 3 to 4 years is probably explained by the higher intraindividual variability, due to random error and change of conditions related to the plasma biomarkers. Bioavailability of carotenoids is influenced by source, concurrent consumption of fat and fiber, and processing, which affects the interpretation of plasma carotenoid concentrations in relation to dietary intake of these compounds.

Although we adjusted for possible nondietary covariates that influence variability (and therefore reliability) over time, it is not possible to obtain 100% reliability. There are inter- and intraindividual sources of biologic variability beyond the control of any study. Additionally, there are errors in the measurements of dietary covariates based on self-report data, residual confounding, and confounding by unmeasured covariates. The Spearman rank correlation showed a high-to-moderate correlation in the ranking of individuals that is better than most biomarkers. The disadvantage of a Spearman correlation is that any systematic error that affects all individuals equally will not be detected. For example, if a change in the laboratory method affected all subsequent assays, we could not detect a difference in ranking based on Spearman correlation. However, mean plasma carotenoid levels were observed to be minimally changed, suggesting no major systematic error.

Multiple factors contribute to poor reliability of a biomarker, including the time of blood collection, laboratory procedures, and storage conditions, as well as biologic and metabolic variability. The use of mixed models to estimate the generalized reliability (including the time variable) mitigates some of these concerns. Taking all these considerations into account, our results suggest good reliability for plasma carotenoid levels that warrant their use in epidemiologic studies, even if collected only once at baseline. Our results regarding reliability of average exposure over time are most relevant when the disease of interest is influenced by exposure over time—for example, cancer, cardiovascular diseases, and diabetes. For diseases triggered by a single episode of exposure—such as an asthma attack—it would be more important to measure exposure at single episode than to estimate the reliability of average exposure.

Reliability of exposure measures has important implications for study design. An assessment method with poor reliability will require a large sample size, and may lead to biased estimates of exposure-disease associations. The estimates of ICC provided here can be used as a basis for estimating the sample size needed to observe associations between plasma carotenoids and diseases of interest. For example, a cohort study on the association of carotenoid levels and breast cancer recurrence would need approximately 375 subjects to detect a 1.5 odds ratio for each standard deviation decrease in carotenoid levels with 90% power,²⁹ given disease prevalence of 20%. However, this assumes that carotenoid values are measured perfectly (ie, ICC = 1). However, based on our data, the ICC range from 0.6 to 0.7 in the best case. Under such conditions, 540 to 630 subjects would be needed to detect the given effect. Thus, not considering the reliability of a measurement tool could lead to substantial overestimates of statistical power.^29-31

In conclusion, reliability is an important aspect of biomarkers. Each of the methods used to measure reliability offers a different perspective on reliability, with mixed models overcoming the limitations of other individual methods. Using the example of plasma carotenoids samples sequentially collected over time, these analyses illustrate the various sources of variability and the impact on study power and sample size.