Analytical and biological variation in repeated sweat chloride concentrations in clinical trials for CFTR modulator therapy

Abstract

Background:

Using sweat chloride as a biomarker for CFTR modifying drugs requires knowledge of analytical and biological variation.

Methods:

979 sweat chloride concentrations from 128 subjects enrolled in the placebo arm of 2 multicenter, investigational drug trials were analyzed to determine coefficients of variation (CV) as well as reference change value (RCV) and index of individuality (II).

Results:

For these populations, calculated values for the two studies were: analytical variation (3.9, 4.1%); within-subject variation (4.4, 6.0%); between-subject variation (8.9, 7.0%); RCV (13.7, 17.0%) and II (0.7, 1.0). Sweat chloride variation was not affected by sex, collection site or sample weight; but was slightly affected by age in one of the two studies.

Conclusion:

Through determination of analytical as well as between- and within-subject variation, and with a larger sample size, our data allows improved estimates of the RCV and II, and can contribute to future trials of CFTR modulators and inform the design and interpretation of n of 1 trials in both research and clinical settings.

1. Introduction

Sweat chloride determinations are used to confirm the diagnosis of cystic fibrosis (CF) and also serve as a biomarker for CFTR function in clinical trials using protein modulation therapy [1-3]. Currently, there is uncertainty surrounding the degree of change in sweat chloride concentrations that reflects a positive response to an investigational drug [2]. Being able to assess a meaningful change is critical to the evaluation of current and future CFTR modifying drugs.

Variation in repeated results for an analyte in a given individual can be attributed to preanalytical, analytical and biological variation [4]. If preanalytical variation can be minimized using standardized collection methods, then this leaves analytical and biological variation to account for the vast majority of change seen in repeated testing. Analytical and biological variation can be used to calculate the reference change value (RCV), also thought of as the medically or clinically significant change between two results for a given analyte [4,5]. The RCV incorporates analytical and within-subject biological variation to allow a probabilistic statement about the significance of change in repeated analysis from an individual. If the change in a repeated test exceeds the RCV, then it can be concluded that a true change has occurred beyond the inherent analytical and biological variation [4]. Knowing the RCV would allow clinicians to objectively evaluate changes in sweat chloride concentrations, e.g., in a given subject’s response to CFTR protein modifying drugs [6]. Another guide to variation, the individuality index (II), has important applications for the use of reference values in laboratory medicine and for the interpretation of test results. The aims of this study were to determine analytical and biological variation, RCV and II of repeated sweat chloride concentrations in 2 multicenter trials of CFTR modulators, using a centralized laboratory for sweat analysis. These data could then inform future multicenter trials of CFTR modulators as well as the design and interpretation of n of 1 trials in research and clinical settings.

2. Methods and materials

This study was a retrospective analysis of repeated sweat chloride tests performed on 128 patients enrolled in the placebo arms of two drugs trials (study 1 and study 2) sponsored by Nivalis Therapeutics Inc. (Boulder, CO) and PTC Therapeutics (South Plainfield, NJ), respectively. Patients enrolled in these trials were recruited from the Cystic Fibrosis Foundation (CFF) Therapeutic Development Network (TDN) sites and had 2 copies of a CF-causing mutation (Table 1). The UNC Office of Human Research Ethics determined the study did not require IRB approval.

Table 1.

Patient demographics and sample characteristics.

		Study 1	Study 2
Timeframe		2013–2015	2009–2011
Number of patients		28	100
Sex	Male	10	52
	Female	18	48
Age	Mean	29 years	24 years
	Range	18–44 years	8–54 years
Genotype		2 copies of F508del	1 nonsense mutation along with a 2nd CF-causing mutationa
Pancreatic status		Insufficient	Insufficientb
Collection timeline		Screening, baseline, days 4, 7, 14,21,28, 42	Screening, week 16, week 32, end of treatment
Number of collection occasions	Total	151	366
	Sweat chloride available from only one arm	5.3% (8/151)	10.4% (38/366)
	Sweat chloride available from both arms	94.7% (143/151)	89.6% (328/366)
Number of samples	Total	292	687
	Left arm	146	346
	Right arm	146	341
Sample weight	Median	57 mg	53 mg
	Range	17–105 mg	15–110 mg
Missing values	Total	10	38
	Not collected/>30 minc	6	20
	QNSd	4	18
	Patients with one QNS	14.3% (4/28)	13.0% (13/100)
	Patients with > 1 QNS	0.0% (0/28)	2.0% (2/100)

^aThere was at least 1 nonsense mutation on all patients and a second CF-causing mutation on the other allele, of which the majority were nonsense/F508del.

^bPancreatic sufficient patients were excluded from the study 2 data.

^cSamples with collection times >30 min were excluded from analysis.

^dQNS is quantity not sufficient and was defined as a sample with volume/weight < 15 μL/15 mg.

Sweat was collected on specified days according to study protocol (Table 1). Samples were collected by trained personnel from both right and left arms with the Macroduct^® Sweat Collection System, frozen (−70 °C) and shipped overnight to the CFF TDN Center for Sweat Analysis Laboratory in Aurora, Colorado for chloride analysis using a digital chloridometer. Sweat collection and analysis were performed in accordance with standardized guidelines [1,7]. All sweat chloride concentrations (right and left arm samples) were included in the analysis when available. Fifteen μL (15 mg) was defined as the minimum acceptable collection volume (weight) and samples with smaller volumes were deemed quantity not sufficient (QNS) and were not analyzed. In addition, pancreatic sufficient patients, and samples that exceeded 30 minutes collection time were excluded.

The collated, de-identified dataset was analyzed using R statistical software, version 3.2.0 (R Core Team, Vienna Austria, 2015). Statistical variance procedures were used to arrive at analytical variation, biological variation (within-subject and between-subject variation), and total individual subject variation of sweat chloride testing for each study. Linear mixed-effects models and likelihood ratio tests were used to calculate variation and provide average total individual subject variation and between-subject biological variation. Total individual subject variation includes preanalytical variation, analytical variation and within-subject variation [9]. Preanalytical variation was minimized due to standardization of collection into Macroduct coils, adherence to established guidelines and study standard operating procedures, training of collection operators, evaluation of storage and shipping effects on sweat stability and performing the analyses at a single, centralized clinical laboratory [7]. Covariates included in the analyses were: age (years), sex, collection site (right or left arm) and sample weight (mg).

Variance components were calculated using standard deviation (SD) and coefficient of variation (CV). Analytical variation was determined by calculating the imprecision of the 100 mmol/L high control (Quantimetrix^®, Redondo Beach, CA) analyzed during the period of time the subjects were tested during each of the studies. The high control was chosen based on its closeness in value to the subjects’ sweat chloride concentrations. Bias (inaccuracy) was calculated using the College of American Pathologists proficiency testing results for the same time period [8]. RCV was calculated using a Z score of 1.65 to reflect a one-tailed 95% probability (p < 0.05) [9]. A Z score for a unidirectional interval was chosen because the interest was in a decrease in sweat chloride concentration. Because biological individuality can affect the use of reference intervals, the index of individuality (II) was also determined. When within-subject variation is less than the between-subject variation, the analyte is said to have individuality and the II can be calculated as the ratio of total within-subject variation to between-subject variation [9].

3. Results

After removing missing values and unsuitable samples, there were a total of 517 collection occasions and 979 sweat chloride samples across both studies. The two study datasets were analyzed separately due to variation in timeframe and subject genotypes (n = 292, study 1; n = 687, study 2) (Table 1). Data were inspected for outliers. Bland-Altman plots and related scatterplots were constructed with the right and left arm values at the same time point and the means of right and left values at two different timepoints as exploratory visualizations (Figs. 1 and S2-S4). Normality was assessed for both studies and the data were sufficiently close to normal to use the typical formulas and to compute IIs and RCVs (refer to online supplement and Fig. S1).

Fig. 1.

Bland Altman plot of the right and left sweat chloride (Cl) concentrations in the two studies. The outer dotted lines represent ±1.96 standard deviations from the mean difference. The mean difference is 0.5 with a standard deviation of 5.7.

Only one covariate of interest, sample weight, had a statistically significant effect on the mean sweat chloride concentration (p = 0.01, study 1; p < 0.001, study 2), with larger sample weights yielding larger sweat chloride concentrations (Fig. 2). The effects of other covariates (age, sex, collection site) on mean sweat chloride concentration were not statistically significant at an alpha of 0.05. In study 1, there were no statistically significant effects on variance for any of the covariates in the multiple regression model, including the weight of the sample (Table 2). Therefore, in study 1, there is not enough evidence that the variation in sweat chloride levels around the mean level changes as a function of any of these variables. In study 2, the variability around the predicted mean varied significantly as a function of sample weight, sex and age (Table 2). Men’s measurements varied by an additional 1.2 mmol/L in absolute value, relative to women’s. The effect of age implied a decrease of 0.08 mmol/L in absolute value with each additional year, and sweat chloride values became less variable by 0.02 mmol/L with each additional mg of sample weight. While sample weight and sex were statistically significant, these amounts were not considered clinically significant. On the other hand, moving from the youngest (8 years) to the oldest subject (54 years) in the study 2 data was associated with a reduction by half of the predicted within-subject variability. As a result, the data were stratified into two age groups (young: <30 years; old: ≥ 30 years) for further analysis. Further variance component analysis, RCV and II are summarized in Table 3 and Fig. 3. Bias (inaccuracy) was computed (0.61%, study 1; −1.3%, study 2) and was determined to be negligible and constant during the study time and therefore was not included in the calculations of analytical variation.

Fig. 2.

Scatterplots of sweat chloride (Cl) concentration against the weight of the sweat sample. For the study 1 model, the squared correlation between fitted and observed sweat Cl values, taking both the sample weight and each subject’s estimated random intercept into account, is 0.71. The slope relating weight to Cl is approximately 0.06. Thus, for a 10 mg (10 μL) increase in weight, sweat Cl concentration would increase by approximately 0.60 mmol/L.

Table 2.

Results of the mixed-effect model for the variance.

Covariate	Study 1			Study 2
	β	χ2	p value	β	χ2	p value
Age	−0.04	1.09	0.30	−0.08	11.0	< 0.001
Sex	0.41	0.48	0.49	1.22	7.3	0.01
Right/left	0.12	0.08	0.77	−0.08	0.1	0.78
Sample weight	−0.01	2.04	0.15	−0.02	3.8	0.05

Table 3.

Sweat chloride measurements of variation, index of individuality and reference change value.

		Study 1	Study 2
			Unstratified	Young	Old
Overall Sample	Mean (mmol/L)	103.4	101.8	102.0	101.1
	SDa (mmol/L)	10.9	10.1	10.1	10.4
Method b	CVcanalytical(%)	3.9	4.1	4.1	4.1
Total individual subject variationc	SD (95% CI d) (mmol/L)	6.1 (5.6–6.6)	7.4 (7.0–7.8)	7.8 (7.3–8.3)	5.9 (5.3–6.7)
	CVtotale (%)	5.9	7.3	7.6	5.9
Within-subject variationd	N	292	687	525	162
	SD (95% CI) (mmol/L)	4.5 (3.8–5.3)	6.1 (5.6–6.6)	6.5 (5.9–7.2)	4.3 (3.2–5.2)
	CVwithinf (%)	4.4	6.0	6.4	4.2
Between-subject variation	N	28	100	78	22
	SD (95% CI) (mmol/L)	9.2 (6.6–11.7)	7.1 (6.0–8.3)	6.6 (5.4–8.0)	8.5 (6.2–11.7)
	CVbetween(%)	8.9	7.0	6.5	8.4
Reference change value (%)g		13.7	17.0	17.7	13.7
Index of individualityh		0.7	1.0	1.2	0.7

^aSD is standard deviation.

^bMethod CV_analytical is the laboratory method imprecision and does not include bias.

^cCV is coefficient of variation.

^dCI is Confidence interval.

^eCV_total was defined as: (${\text{CV}}_{\text{analtyical}}^2$ + ${\text{CV}}_{\text{within}}^2$)½.

^fCV_within was calculated as (${\text{CV}}_{\text{total}}^2$ − ${\text{CV}}_{\text{analytical}}^2$)½

^gRCV was calculated using the formula: 2^½ * Z * (${\text{CV}}_{\text{analytical}}^2$ + ${\text{CV}}_{\text{within}}^2$)1^/2, where a Z score of 1.65 was chosen to reflect a one-tailed 95% probability (p < 0.05) [9].

^hII was calculated as the ratio of total within-subject variation to between-subject variation [${\text{CV}}_{\text{analytical}}^2$ + ${\text{CV}}_{\text{within}}^2$]^½/CV_between [9].

Fig. 3.

Components of variation.

4. Discussion

The importance of understanding variability in sweat chloride determinations has recently been underscored [11]. In addition, analysis of the variation of sweat chloride determinations in additional placebo groups from clinical trials has been called for [12]. Herein we have analyzed sweat chloride variability from two additional clinical trials and have added analytical variability to within- and between-subject determinations for the first time in a clinical trial setting.

Using biomarkers in serial studies, a change in a repeated value in an individual is attributed to preanalytical, analytical and within-subject biological variation. Ideally these 3 contributors to variation would be small so that any change would be attributed to an intervention, or change in health or disease. In this study, preanalytical variation was minimized through standardization so the change in repeated sweat chloride values was due to analytical and biological variation.

Studies examining the reproducibility of repeated sweat chloride concentrations for the diagnosis of CF have been limited in terms of subject numbers, use of nonstandard methods of collection and analysis and non-inclusion of analytical variation [13-16]. Recent publications on sweat chloride results have focused on its use as a robust biomarker for determining efficacy of CFTR protein modulating drug therapy [1,3,12,17-19]. Various levels of decrease in sweat chloride concentration in the treatment arm compared to the placebo arm have been observed as indicating a favorable response to CFTR modulators. These range from a decrease of 9.1 mmol/L; 11.8 mmol/L;15 mmol/L; or 29–47% from baseline [1,12,17-19]. In a clinical trial with Ivacaftor in subjects with at least one copy of the G551D mutation, the placebo subjects within-subject SD was 4.4 mmol/L, similar to the within-subject SD found here (4.3–6.5 mmol/L) [1]. In another study examining the variation in sweat chloride concentrations in patients with G551D mutation, their between-subjects SD (8.9 mmol/L) was similar to our subjects SD (9.2 mmol/L) for patients homozygous for the F508 del mutation [12]. However, their within-subject SD (8.1 mmol/L) differed from the value found here (4.5 mmol/L). It is possible that the disparity is due to a difference in CFTR gene mutation, but may also be because the within-subject values reported here have been corrected for analytical variation while the other study did not appear to do so [12].

In this population, mean sweat chloride concentration was slightly affected by sweat weight (Fig. 1); findings which have been confirmed by others and most likely are not clinically significant [12]. In study 2, the variability of sweat chloride around the predicted mean varied significantly as a function of sample weight but the amount was not considered clinically important. Because sample weight (≥ 15 mg) does not play a very large role in sweat chloride variation, there is no reason to treat the sweat chloride concentration from the sample with greater weight (or volume) as the more reliable sample. Therefore, if only one chloride value is to be reported from bilateral testing instead of the preferred two values, it would be better to average the chloride concentrations instead of selecting the chloride concentration from the site with the greater sweat weight or volume.

Sweat chloride variation was not significantly affected by age, sex, collection site (right or left arm) or sample weight in study 1 which supports the use of a single reference interval for both males and females. Although variation did differ by sex in study 2, the difference was not considered clinically significant. Differences in sweat chloride variation due to age were explored by stratifying the data of study 2 into younger and older age groups. Within-subject variation and the RCV for the younger group were higher than the older group indicating that there may be more variability in the sweat chloride of younger patients compared to older patients. This association was not detected in study 1 presumably because the sample size in study 1 was smaller (28 vs. 100 patients), and the range of ages was narrower (18–44 vs. 8–56 years). Thus, the effect of age may need to be considered when evaluating sweat chloride in clinical trials.

A number of factors go into evaluating efficacy of treatment with CFTR modulators. While sweat chloride change does not correlate on the individual level with FEV1 (the most commonly used outcome measure for pulmonary benefit), sweat chloride change does correlate with FEV1 on the population level, especially when multiple trials are considered [20-22]. In either case, sweat chloride change is of interest in establishing CFTR activity of a given treatment. Considering the effect in an individual, as for example in an n of 1 trial, a change in sweat chloride concentration indicating a favorable response could be arbitrarily assigned. The process could be given statistical rigor, however, through calculation of the RCV. The RCVs for the populations studied here ranged from 13.7–17.7% (Table 3). Given the results in these clinical trials, if these placebo patients were to receive a CFTR modifying drug, their sweat chloride would have to decrease by > 13.7–17.7% from baseline to indicate a change beyond the usual sweat chloride analytical and biological variation. It can also be surmised that because of the large RCV found in this study, the treatment effect for a drug intervention would also have to be large in order for it to be a clear change in the value [12]. It is important to note that the various RCVs are specific for these populations of pancreatic insufficient individuals with given mutations and to the laboratory performing the analysis and may not be generalizable to all CF patients in clinical trials. For example, pancreatic sufficient patients may have greater biological variation of sweat chloride and other laboratories may have different values for their analytical variation.

In clinical practice it is useful to apply the concept of RCV to repeated diagnostic testing. For example, a physician wishes to know whether a patient’s glucose has changed in a meaningful way from the previous year from 101 mg/dL to 134 mg/dL. Assuming similar preanalytical and analytical conditions, the RCV for random (non-fasting) glucose is 21.1% [10]. The difference between the repeated tests in this example was 32.7% (which is greater than the RCV of 21.1%), which indicates a meaningful increase. Likewise, when two sweat chloride results are different and it appears that there is a change in concentration, the RCV can be useful for example, if a patients sweat chloride concentration had changed from 100 mmol/L to 80 mmol/L (equivalent to 20%). This amount of change is greater than the RCV for sweat chloride determined here for this population and would be interpreted as a meaningful change in a patient’s sweat chloride. Because the change exceeds the RCV it reflects a change beyond inherent analytical and biological variation in sweat chloride.

It is important to note that we found that analytical variation is not negligible in computing the RCV and total variation. There is no established target for analytical variation for sweat chloride testing as there is for serum chloride; however, it is suggested for a given clinical laboratory analyte that the desirable analytical variation be half of the within-subject variation [5]. If laboratories can lower the imprecision of sweat chloride analysis, then the RCV will decrease and small changes in repeated analytes will be more informative [9]. Because analytical variation and bias affect RCV, it is especially important that laboratories attend to quality management practices and try to minimize their effects. If laboratories lower the imprecision of sweat chloride analysis, it could decrease the RCV. For example, if the analytical variation in the unstratified study 2 patients was lowered by half from 4.1% to 2.0%, the RCV decreases from 17.0% to 14.7%. In this study, decreasing the analytical variation by 50% decreases the RCV by 13.5% because of the large influence of within-subject variability in this population. Efforts to reduce the analytical variation of sweat chloride testing include: the adherence to standardized procedures for collection and analysis, mandatory and ongoing competency assessment of laboratory technologists and research coordinators, direct and frequent observation of performance by laboratory supervisors knowledgeable of the testing procedures, and use of validated chloridometers [7]. (See the online supplement for a discussion about efforts to reduce variation in common laboratory analytes such as lipids and glycohemoglobin.).

In these subjects the II ranged from 0.7 to 1.2. Individuality indices are interpreted by using predetermined set points of 1.4 and 0.6 [4]. An II value for a given analyte of > 1.4 is unusual and signifies a low degree of individuality while an II < 0.6 is more common and indicates a high degree of individuality [4,9,23]. An analyte with an II > 1.4 means that a repeated value could distribute anywhere within the reference range for that population [4,9]. Using the example of a random serum glucose, the within-subject variation is much less than the between-subject variation and the II is 0.6 [10]. This means that an individual’s repeated glucose values will fall within a small range compared to the larger reference range for random serum glucose of 70–140 mg/dL [24]. As expected, in this study of sweat chloride values, the within-subject variation was less than the between-subject variation and so the indices indicate that sweat chloride has a moderate degree of individuality. This means that an individual’s sweat chloride stays within a narrower range compared to the wider diagnostic range for CF of 60–165 mmol/L. In other words, a CF patient could have a sweat chloride value that is very unusual for them, but still lies within the diagnostic range. The II of 0.7 supports the use of repeated sweat chloride determinations for diagnosis, especially in cases of CFTR-related metabolic syndrome and the use of serial determinations of sweat chloride in order to detect a response to a treatment regime (such as an n = 1 trial of an investigational drug).

Our results could also inform power analyses/required sample sizes for future studies that use sweat chloride as the response. We cannot specify the sample sizes that would be needed for future studies, because that would depend on specific design choices of the planned study and the effect size the researchers would want to detect. However, if investigators are willing to assume their patients will be similar to the ones analyzed here, our estimated SDs could be used. The investigators would need to specify the mean difference they would like to be able to differentiate from 0. Then, this amount can be divided by our observed SDs to compute the effect size. For example, if the study is conducted between patients, all sources of variation (analytical, within-subject, and between-subject) would be present. The weighted average of the SD from study 1 and the unstratified SD from study 2 is 10.3. Thus, for example, if researchers wanted to detect a 5 mmol/L change in a between-subjects trial with a single measurement occasion, they would need 68 patients per arm to achieve 80% power [25]. On the other hand, if the study would be within patients, our total individual subject variation (analytical plus within) would be used. The weighted average of these SDs from study 1 and the unstratified version of study 2 is 7.1.

The strengths of this study lie in the total number of sweat chloride results (979) taken from at least 3 sets of repeated tests; the minimization of preanalytical variation; the inclusion of analytical variation in the determination of biological variation and the examination of different mutations. The limitations include the limited generalizability due to the influence of a specific laboratory’s analytical variation on the RCV. Work is needed to reduce analytical variation in sweat chloride analysis to include more investigation into interlaboratory differences using duplicate samples. Some data examining other mutations (G551D CFTR mutation) exists, but more studies are needed for consensus [16]. Future studies should incorporate more CF patients including infants, children, and individuals with different CFTR gene mutations.

5. Conclusion

This study represents a novel approach to determining biological sweat chloride variation in CFTR protein modulator trials because it incorporates analytical variation into the calculations. As more data become available from future studies, it is possible that a standardized unit of biological variation for sweat chloride could be determined. Then, if laboratories minimize preanalytical variation and maintain desirable analytical variation, a robust and transferable RCV for sweat chloride concentrations can be established which has the potential of lowering the number of subjects enrolled in clinical trials.