Reproducibility and Variability of Protein Analytes Measured Using a Multiplexed Modified Aptamer Assay

Abstract

Background:

There is growing interest in the use of multiplexed aptamer-based assays for large-scale proteomic studies. However, the analytic, short- and long-term variation of the measured proteins is largely uncharacterized.

Methods:

We quantified 4001 plasma protein analytes from 42 participants in the Atherosclerosis Risk in Communities (ARIC) Study in split samples and at multiple visits using a multiplexed modified aptamer assay. We calculated the CV, Spearman correlation, and intraclass correlation (ICC) between split samples and evaluated the short-term (4–9 weeks) and long-term (approximately 20 years) variability using paired t-tests with log-transformed protein concentrations and Bonferroni-corrected significance thresholds. We performed principal component (PC) analysis of protein analyte concentrations and evaluated their associations with age, sex, race, and estimated glomerular filtration rate (eGFR).

Results:

The mean baseline age was 57 years at the first visit, 43% of participants were male and 57% were white. Among 3693 protein analytes that passed quality control, half (n = 1846) had CVs < 5.0%, Spearman correlations > 0.89, and ICCs > 0.96 among the split samples. Over the short term, only 1 analyte had a statistically significant difference between the 2 time points, whereas, over approximately 20 years, 866 analytes (23.4%) had statistically significant differences (P < 1.4 × 10⁻⁵, 681 increased, 185 decreased). PC1 had high correlations with age (−0.73) and eGFR (0.60). PC2 had moderate correlation with male sex (0.18) and white race (0.31).

Conclusions:

Multiplexed modified aptamer technology can assay thousands of proteins with excellent precision. Our results support the potential for large-scale studies of the plasma proteome over the lifespan.

Circulating concentrations of specific proteins are routinely measured in clinical research and for medical diagnosis and prognosis. High-throughput identification and quantification of proteins with multiplexed technology provide opportunities for mining the human proteome to improve our understanding of disease processes, link genetic variants to blood protein regulatory networks, and potentially discover novel biomarkers for early detection and disease diagnosis. Historically, immunoaffinity-based methods that target a single protein are used most commonly for clinical assays (1). Cross-reactivity of affinity reagents limits the number of proteins that can be evaluated in a multiplexed immunoaffinity-based assay to the low hundreds (2). Single-stranded DNA aptamer-based assays have been developed as an alternative to immunoaffinity-based methods (3) and have been increasingly applied in clinical and epidemiological studies (4–8). For example, a panel of 9 proteins identified from thousands has been shown to predict cardiovascular disease better than traditional risk factors (4).

The human proteome is dynamic and sensitive to external stimuli. There may be considerable short-term (i.e., weeks) and long-term (i.e., years) variation in addition to sample handling and laboratory (methodologic) variability. Understanding the laboratory reproducibility of aptamer-based assays, as well as short-term and long-term biological variation of proteins, is critical for designing longitudinal studies of the proteome. Previous studies have reported overall intra- and interassay median CV of 4% to 8% for multiplexed aptamer-based assays (4, 6, 9). Concentrations of specific proteins may vary by age, race, and sex and be influenced by the function of the kidney, which filters, reabsorbs, and catabolizes many low molecular weight proteins (10). Few studies have investigated short- and long-term variation of a large number of plasma proteins along with laboratory reproducibility. We conducted a study to assess the laboratory reproducibility, short-term, and long-term biological variation in >3500 plasma protein analytes measured by a multiplexed modified aptamer assay. The plasma samples were collected at multiple time points from 42 participants selected from the Atherosclerosis Risk in Communities (ARIC) Study using stratified sampling based on race, sex, and kidney function.

METHODS

Study population

The ARIC study is a prospective cohort study of 15792 adults (45–64 years old at visit 1 in 1987–1989) recruited from 4 US communities: Forsyth County, North Carolina; Jackson, Mississippi; suburban Minneapolis, Minnesota; Washington County, Maryland (11). For the purpose of this study, we randomly selected 42 participants from those with data at visits 2 and 5, and the repeated visit 5 from 7 strata (6 participants each) defined by race (self-reported white or black), sex (male or female), and estimated glomerular filtration rate (eGFR) at visit 5 (30–59 mL/min/1.73m² or 60–100 mL/min/1.73m²; see Table 1 in the Data Supplement that accompanies the online version of this article at http://www.jalm.org/content/vol4/issue1). The sample size of 42 was determined on the basis of having 80% power to detect a minimum Spearman correlation of 0.43 at an α concentration of 0.05 for a 2-sided test. eGFR concentration was calculated from serum creatinine after calibration across visits using the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation (12, 13). Protein analyte levels from these 42 participants were measured using previously unthawed EDTA plasma collected at visit 2 (1990–1992), visit 5 (2011–2013), and a repeated visit 5 (4–9 weeks after visit 5). Plasma samples collected during the repeated visit 5 were split as 2 vials for assessing technical reproducibility with their identities masked to personnel processing the modified aptamer assay. The ARIC protocol for blood sample collection and processing was designed to minimize spontaneous biochemical reactions after blood collection and is consistent with recommended practice for epidemiological studies (14, 15). In brief, blood samples were put immediately in an ice water bath after venipuncture. Centrifugation was then performed within 10 min after venipuncture at room temperature (15–25 °C). After centrifugation, the aliquots were stored at −80 °C within 90 min from venipuncture.

Modified aptamer protein assay, quality control, and normalization

Concentrations of protein analytes were quantified using a multiplexed modified DNA-based aptamer technology (SOMAscan assay), which transforms individual protein concentrations into a corresponding modified aptamer (SOMAmer reagent) concentration that can be quantified by standard DNA microarrays in relative fluorescence units (RFUs) (16, 17). The SOMAscan assay included 4001 aptamers that mapped to 3581 unique proteins or protein complexes in the Universal Protein Resource (UniProt) databases. Our unit of analysis included the measures of protein analyte from each aptamer quantified in RFUs.

Protein analyte measurements underwent the regular SOMAscan data standardization and normalization process (18). Briefly, hybridization control normalization was first applied to each sample based on a set of hybridization control sequences to correct for systematic biases during hybridization. Second, median signal normalization was applied to measures within a plate to remove sample or assay biases that may be because of pipetting variation, variation in reagent concentrations, assay timing, and other sources of systematic variability within a single plate run. Finally, each plate contained calibrator samples for each SOMAmer reagent, which was used to correct for plate-to-plate variation based on established global reference standards. Protein analytes with calibration factor greater or less than the median calibration factor (0.4) were excluded from all analyses.

Traditional protein assays

Measurements of 9 proteins assessed in the multiplex aptamer-based assay were previously quantified using clinical assays in ARIC (9 were available at visit 2 and 8 at visit 5): albumin (bromcresol green colorimetric assay), alanine aminotransferase (ALT; a kinetic reaction assay), β₂-microglobulin (B2M; an immunoturbidimetric assay), high-sensitivity C-reactive protein (CRP; an immunoturbidimetric assay), cystatin C (an immunoturbidimetric assay), N-terminal pro-B-type natriuretic peptide (NT-proBNP; a sandwich immunoassay), parathyroid hormone (PTH; a sandwich immunoassay), high-sensitivity cardiac troponin T (hs-cTnT; an immunoassay), and thyroid-stimulating hormone (TSH; a sandwich immunoassay). All clinical assays were conducted using Roche Diagnostics instruments, except for CRP at visit 5, which used a Beckman Coulter instrument. In contrast to the multiplexed assays, all of the clinical assays used serum, except for hs-cTnT at visit 5, which was measured in plasma. All clinical assays had CVs < 10%. Details are reported in Table 2 in the online Data Supplement.

Analysis of reproducibility.

To verify the split sample identity, we performed hierarchical clustering of the protein analyte measures using Ward’s distance and compared the Spearman correlation coefficients between the protein analyte measures from the split samples vs those from 2 randomly selected samples.

To quantify reproducibility of the protein measures, we computed the CV, Spearman correlation, and intraclass correlation (ICC) of the measures of protein analytes between the split samples. While CV and ICC quantify the magnitude of the variation between the split samples, Spearman correlation compares the ranking of the measures between the split samples. CV was defined as $\text{exp}\left(\sqrt{mean\left(variance\left(log\left(RFU\right)\right)\right)}\right)-1$ (19). Compared with the common definition of CV (average of the SD divided by the mean of blind duplicate samples), this alternative definition avoids underestimation of the CV when the SD of the blind duplicate samples is dependent on the mean. We also evaluated CV stratified by eGFR at 60 mL/min/m². The 95% CI for the Spearman correlation was estimated using the Bonett and Wright method (20). ICC was estimated using 1-way ANOVA (21).

To evaluate reproducibility across platforms, we computed the Spearman correlation coefficients for 9 proteins that were measured by both clinical assays and the modified aptamer assay at visit 2 and 8 proteins at visit 5. For the 8 proteins with available measures from clinical assays at both visits 2 and 5, we computed change from visit 2 to visit 5 using values from the clinical assays and the modified aptamer assay, then computed Spearman correlation coefficient between the changes obtained from the clinical assays and the modified aptamer assay.

Analysis of short- and long-term variability

To reduce skewness, log base 2 transformation was applied for the analysis of variability. Statistical significance was determined using the Bonferroni method. To evaluate short-term variability, we compared the measures of the protein analytes at visit 5 with those at the repeated visit 5 (4–9 weeks after visit 5). Given that 2 measures were available for each protein analyte at the repeated visit 5 from the reproducibility study, we randomly selected 1 of the 2 measures to be included in the dataset for short-term variability analysis. To evaluate long-term variability, we used the measures of the protein analytes at visits 2 and 5 (approximately 20 years apart). To contrast assay reproducibility in split samples with biological short- and long-term variability, we evaluated whether the paired measures between split samples, in the short term and long term, differed significantly using paired t-tests. To investigate whether heterogeneity in protein analyte concentrations might have changed over the long term, we tested for the difference in variance between visits 2 and 5 for the paired protein concentrations (22).

To explore the extent to which long-term variation may be related to age, sex, race, and kidney function, we performed principal component analysis (PCA) using the measures at visits 2 and 5 and computed the Spearman correlation coefficients of the first 10 principal components (PCs) with these 4 demographic and kidney function variables. All analyses were conducted using R version 3.4.1.

RESULTS

Participant characteristics and summary of protein analytes

Among the participants, 43% were male and 57% were white. The mean age at visit 2 (1990–1992) was 57 years, and the mean age at visit 5 (2011–2013) was 76 years (Table 1). Of the 4001 protein analytes, 92% (3693) passed the calibration filter and were included in our analyses. These 3693 protein analytes mapped to 3323 proteins or protein complexes in UniProt, of which 90.3% (3001) were tagged by 1 modified aptamer. Three were human virus proteins.

Table 1.

Participant characteristics.^a

Variable	Visit 2 (1990–1992)	Visit 5 (2011–2013)
Male, n (%)	18(42.9)
White, n (%)	24(57.1)
Age, mean (SD)	56.6(4.8)	77.5(5)
BMI,b mean (SD)	28.6(5.5)	30.2 (5.2)c
Smoking, n (%)
Current smoker	3(7.1)	1 (2.4)d
Former smoker	11 (26.2)	16 (38.1)d
Never smoker	28(66.7)	23 (54.8)d
SBP, mean (SD)	118.2(17.9)	134.5(21.9)
DBP, mean (SD)	72.2(10.8)	66.2(8.8)
Hb A1c, median (1st, 3rd quartile), %	5.5 (5.2, 5.8)	6.0 (5.6, 6.6)
Total cholesterol, mean (SD)	203.4(26.6)	173.1 (30.3)
Diabetes, n (%)	6(14.3)	18 (42.9)c
Hypertension, n (%)	15 (35.7)	29(69.0)
Hypertension medication, n (%)	12(28.6)	34(81.0)
Cholesterol lowering medication, n (%)	4(9.5)	28(66.7)
eGFR, mean (SD), mL/min/1.73m2	96.4(11.8)	64.8(13.7)

^aSmoking status was self-reported. HbA1c was measured using NGSP-certified instruments (Tosoh Corporation) standardized to the Diabetes Control and Complications Trial (13). Diabetes was defined as a fasting glucose of ≥126 mg/dL or a nonfasting glucose of ≥200 mg/dL, self-reported of physician diagnosis of diabetes, or taking diabetes medication during the 2 weeks prior to a study visit. Hypertension was defined as the mean of 3 sitting measurements of systolic blood pressure of ≥140 mmHg, of diastolic blood pressure of ≥ 90 mmHg, or self-reported use of hypertension medication during the 2 weeks prior to study visit. Hypertension medication and cholesterol lowering medication were based on the inspection of medication containers during study visit.

^bBMI, body mass index; SBP, systolic blood pressure; DBP, diastolic blood pressure; HbA1c, hemoglobin A1c; NGSP, National Glycohemoglobin Standardization Program.

^cn = 41.

^dn = 40.

Split sample reproducibility of protein analytes from aptamer-based assay

We limited the analysis to 40 of the 42 split sample pairs (2 pairs were excluded because of suspected mislabeling as evidenced by mispairing in a nearest neighbor dendrogram; see Fig. 1 in the online Data Supplement). Among the 40 split sample pairs, the median (25th and 75th percentile) of the reproducibility metrics was as follows: CV, 5.0 (4.1, 6.9); ICC, 0.96 (0.92, 0.98); and Spearman correlation coefficient, 0.89 (0.81, 0.94; Fig. 1). Overall, the 3 metrics yielded consistent results (see Fig. 2 in the online Data Supplement). For example, of the 3287 protein analytes with CVs < 10%, 96.7% had high ICC (>0.80) and 81% had high Spearman correlation coefficients (>0.80). Table 3 in the online Data Supplement presents these reproducibility metrics (CV, ICC, and Spearman correlation coefficients) of all 1830 protein analytes with CV < 5%. Among the split samples from participants with eGFR below and ≥60 mL/min/m², the CVs were similar—median CV (25th and 75th percentile), eGFR < 60 mL/min/m², 5.0 (4.1, 6.9), and eGFR ≥ 60 mL/min/m², 4.9 (4.0, 6.4).

Fig. 1.

Measures of the reproducibility of 3693 protein analytes in 40 split-sample pairs.

Compared with clinical assays, 6 of the 9 proteins with available clinical measures had high correlations with measures from the modified aptamer assay at visit 2: CRP, TSH, cystatin C, B2M, PTH, and NT pro-BNP (Spearman correlations >0.8; Table 2). Five of these 6 proteins also had high correlations (Spearman correlations > 0.8) at visit 5, and clinical measures for the sixth (PTH) was not available at visit 5. The long-term changes of these 5 proteins measured from clinical assays were also highly correlated with the change in these 5 proteins as measured by the modified aptamer assay (Spearman correlations > 0.75).

Table 2.

Spearman correlations of protein measures quantified from modified aptamer assay and other clinical assays.^a

	Visit 2		Visit 5		Change (log scale) between visits 2 and 5
Protein	N	Spearman correlation (95% CI)	N	Spearman correlation (95% CI)	N	Spearman correlation (95% CI)	Aptamer platform CV, %	Clinical assay CV, % visit 2, visit 5
hs-CRP	39	0.96(0.91,0.98)	42	0.97(0.93,0.98)	39	0.90(0.80,0.95)	11.3	4.3,2.2
TSH	39	0.93(0.86,0.97)	42	0.94(0.87,0.97)	39	0.94(0.88,0.97)	4.1	<10,3.6
Cystatin C	40	0.89(0.77,0.95)	42	0.91 (0.81,0.95)	40	0.79(0.61,0.9)	15.2	3.1,3.1
B2M	39	0.87(0.74,0.94)	42	0.92(0.83,0.96)	39	0.76(0.55,0.88)	17	7.3,7.3
PTH	39	0.85(0.71,0.93)	–	–	–	–	10.2	5.1,–
NT-proBNP	35	0.84(0.67,0.92)	40	0.81 (0.64,0.9)	33	0.80(0.59,0.91)	3.9	5.4,7.0
Albumin	39	0.56(0.27,0.75)	42	0.49 (0.2,0.7)	39	0.28 (−0.05, 0.55)	4.2	4.0,4.0
ALT	39	0.53(0.24,0.73)	42	0.36(0.05,0.6)	39	0.38(0.06,0.63)	7.2	5.6,5.6
hs-cTnTb	15	0.52 (−0.02, 0.83)	42	0.46(0.16,0.68)	15	0.14(−0.4, 0.61)	6.3	6.0,6.4

^aWhen multiple CVs were reported from ARIC clinical assays (low and high concentrations or intraday and interday runs; see Table 2 in the online Data Supplement), we selected the larger CV for this table.

^bLimited to values above lower limit of detection (LOD) from the targeted (clinical) hs-cTnT assay (Roche Diagnostics).

Difference between split samples and short- and long-term variabilities

None of the protein analytes had statistically significant differences in concentrations between the split samples. Over the short term (4–9 weeks), only 1 protein analyte had a statistically significant difference in concentrations (P < 1.35 × 10⁻⁵ = 0.05/3693; see Table 4A in the online Data Supplement). Over the long term (approximately 20 years), 866 protein analytes (23.4%) had statistically significant differences in concentrations: 681 (18.4%) increased and 185 (5.0%) decreased (see Table 4B in the online Data Supplement). The median fold change (25th and 75th percentiles) was 0.52 (0.33, 0.82) among those with a significant increase in concentrations, and −0.37 (−0.60, −0.25) among those with a significant decrease in concentrations. The protein analytes with statistically significant increases included the following low-molecular-weight kidney function biomarkers: B2M (fold change, 0.56, P = 4.6 × 10⁻¹⁷) and cystatin C (fold change, 0.38, P = 1.1 × 10⁻¹³). Fig. 2 presents the gaussian kernel smoothed distributions of the differences between the split samples and the fold changes in the short and long terms. Fig. 3 in the online Data Supplement presents the scatter plots of measures of the top 10 protein analytes with the most statistically significant long-term changes. Regarding the difference in variance over the long term, 284 protein analytes (7.7%) had a significant change: 163 (4.4%) increased and 121 (3.3%) decreased (see Table 4C in the online Data Supplement). The top PCs generated from the measures of protein analytes from visits 2 and 5 had moderate-to-high correlations with key demo-graphic and clinical variables. PC1 had high correlations with both age (−0.73) and eGFR (0.60; see Fig. 4 and Table 5A in the online Data Supplement). PC2 had moderate correlations with male sex (0.18) and white race (0.31). The 50 protein analytes with the highest absolute loadings in PC1 and PC2 are reported in Table 5, B and C in the online Data Supplement.

Fig. 2.

Gaussian kernel smoothed distribution of the difference of the protein analytes between time points: split samples vs short term (6–8 weeks apart) in (A) (n = 40) and short term vs long term (~20 years) in (B) (n = 42).

DISCUSSION

In this study, we evaluated the technical reproducibility and the short- and long-term biological variability of 3693 protein analytes measured by a commercial multiplexed modified aptamer assay (SOMAscan). Measures of the majority of the protein analytes were reproducible, demonstrated by split sample CVs of <10%. Compared to measures of 9 proteins quantified using clinical assays, 6 proteins were highly correlated across platforms, including long-term changes for 5 proteins: CRP, TSH, cystatin C, B2M, and NT pro-BNP. Over the short term, only 1 protein analyte had a statistically significant change, although, over the long term, almost a quarter of the protein analytes (23.4%) had a statistically significant change. The long-term variation of the protein analytes as captured by PCs was correlated with key demographics and clinical variables. These results suggest stability in the protein analytes over the short term and the ability to quantify longer term changes in studying the proteome as measured by this multiplexed modified aptamer technology.

The high reproducibility of this multiplexed modified aptamer assay in our study is consistent with the results from previous studies (4, 6, 9, 23, 24), which reported median CVs of 4% to 8%. Few prior studies have evaluated measures of the protein analytes over time and across platforms (6, 25–27). We compared short-term (4–9 weeks) and long-term (approximately 20 years) changes in the same individuals across platforms. Our comparison included important biomarkers of chronic diseases and hormones: CRP, an acute-phase reactant; NT-proBNP, a marker of heart failure; cystatin C and B2M, markers of kidney function; and TSH and PTH, 2 key hormones involved in multiple metabolic processes (28–30). Three proteins, albumin, ALT, and hs-cTnT, had only moderate correlations between the clinical assays and the modified aptamer assay, suggesting that, for some proteins, the established and the modified aptamer methods may differ meaningfully with implications for estimating disease associations. Over the long term, substantially more protein analytes increased (18%) rather than decreased (5%). The increase in protein concentrations over the long term may be partly attributed to reduced kidney function in older age because the kidney plays an important role in the removal of low-molecular-weight proteins (31, 32). In aggregate, the first PC generated by protein measures over the long term was also negatively correlated with eGFR, a measure of kidney function.

Our study included community-dwelling adults with a range of demographic and clinical characteristics. We evaluated reproducibility using multiple metrics, including a comparison of long-term changes between 2 assay platforms. Nonetheless, some limitations of the study warrant mention. The number of proteins compared with clinical assays was limited. We sought to quantify the overall pattern of reproducibility and variability instead of making inference on any specific protein analyte. Hence, we designed a study of moderate sample size. The split-sample assays were conducted in 1 day on 2 plates and therefore we only quantified intraday, rather than interday, laboratory reproducibility or drift. Our experimental design could not exclude the possibility of protein degradation during long-term storage. The overall low proportion of protein analytes having significant change in variance between visits 2 and 5 (7.7%) did not support widespread protein degradation. Studies of stability of individual proteins among the thousands quantified after long-term storage will be useful. Further, the modified aptamer technology provides relative quantification instead of absolute quantification, and the specificity of each modified aptamer may need to be validated in future studies.

In conclusion, in this study of 3693 protein analytes at 3 time points over 20 years, we found a commercial modified aptamer-based technology to be highly reproducible. These results support the use of this technology in clinical and epidemiologic studies of the proteome for identifying disease determinants and estimating disease risk.

IMPACT STATEMENT.

Proteomic assay has increasingly been used in epidemiological studies. We quantified plasma proteins from 42 individuals at multiple time points using a modified aptamer assay and assessed reproducibility, short- and long-term variation. Among 3693 protein analytes, half had CV <5.0% between split samples. Over 4–9 weeks, 1 analyte had a statistically significant difference (P < 1.4 × 10⁻⁵) vs 23.4% over approximately 20 years. Principal component (PC) 1 had high correlation with age (−0.73) and estimated glomerular filtration rate (0.60). Multiplexed modified aptamer technology can assay thousands of proteins with excellent precision, supporting the potential for large-scale longitudinal studies of the plasma proteome.

Nonstandard abbreviations:

ARIC

Atherosclerosis Risk in Communities

eGFR

estimated glomerular filtration rate

CKD-EPI

Chronic Kidney Disease Epidemiology Collaboration

RFU

relative fluorescence unit

ALT

alanine aminotransferase

B2M

β₂-microglobulin

CRP

C-reactive protein

NT-proBNP

N-terminal pro-B-type natriuretic peptide

PTH

parathyroid hormone

hs-cTnT

high-sensitivity cardiac troponin T

TSH

thyroid-stimulating hormone

ICC

intraclass correlation

PCA

principal component analysis

PC

principal component