PERFORMANCE OF SUCCINYLACETONE ASSAYS AND THEIR ASSOCIATED PROFICIENCY TESTING OUTCOMES

Abstract

Background

Succinylacetone (SUAC) is the primary metabolic marker for hepatorenal tyrosinemia.

Materials and Methods

We used results reported for dried-blood-spot proficiency testing (PT) specimens and hepatorenal tyrosinemia patients’ newborn screening (NBS) samples to demonstrate analytic biases in SUAC recoveries and differences in presumptive clinical classifications.

Results

SUAC recoveries from non-kit and NeoBase™ kit tandem mass spectrometry methods were markedly different. Kit users that set high cutoff values submitted discordant clinical assessments of “within normal limits” for PT specimens enriched with 10–15 µmol SUAC/L blood. SUAC levels in tyrosinemia patients’ NBS samples analyzed by NeoBase™ kit were lower than those in samples analyzed by non-kit methods.

Conclusions

Analytic biases in SUAC recoveries were consistent from 2009–2011. Discordant clinical assessments of PT specimens were associated with high cutoff values for NeoBase™ kit results. Method-related differences in SUAC concentrations of tyrosinemia patients’ samples were consistent with those of PT specimens.

INTRODUCTION

Hepatorenal tyrosinemia (tyrosinemia type I) is an inborn error of metabolism that, if untreated, can cause death from liver failure in the early years of life [1]. Hepatorenal tyrosinemia newborn screening (NBS) tests based on tyrosine elevation lack specificity because neonatal hypertyrosinemia also occurs in other physiological and pathological conditions and lack sensitivity because tyrosine levels are not consistently elevated in affected newborns [2–4]. Succinylacetone (SUAC) is accumulated in the blood of newborns with hepatorenal tyrosinemia and is specific for that disorder [5]. The development of tandem mass spectrometry (MS/MS) NBS methods for measuring SUAC concentration [2,4,6] has produced higher specificity of measurements, resulted in reduced risk of false-positive and false-negative reports, and achieved better acceptance of hepatorenal tyrosinemia in NBS panels.

In January 2008, the Newborn Screening Quality Assurance Program (NSQAP) of the Centers for Disease Control and Prevention (CDC) initiated a pilot SUAC proficiency testing (PT) program to support laboratories that perform NBS tests for hepatorenal tyrosinemia using the SUAC marker, and later in that year, NSQAP added SUAC to its routine PT programs [7].

We used data and screening program practices reported by NSQAP PT program participants in 2009, 2010, and 2011 to compare their measured SUAC concentrations, their cutoff values used to classify test results as presumptive-positive or negative, and their presumptive clinical classifications of NSQAP’s PT specimens. Additionally, we compared SUAC levels found in NBS samples from confirmed hepatorenal tyrosinemia patients screened by derivatized tandem mass spectrometry (MS/MS) non-kit methods with those of a similar number of patients screened by the non-derivatized MS/MS NeoBase™ kit (www.PerkinElmer.com) method. In this report, we describe observed patterns of analytic biases and account for their effects on the presumptive clinical classifications of dried-blood-spot (DBS) PT specimens.

2. MATERIALS AND METHODS

2a. Preparation and distribution of dried-blood-spot (DBS) PT materials

All DBS materials for NSQAP’s PT program were made from whole blood units collected in citrate phosphate dextrose adenine (CPDA-1), purchased from a regional blood bank, and adjusted to 50% hematocrit by plasma removal. Specimens in the PT panels either contained endogenous SUAC levels (< 0.7 µmol SUAC/L blood) or were enriched with predetermined quantities of aqueous SUAC standard solutions prepared from SUAC (4,6-dioxoheptanoic acid, 99.5% purity) from Sigma-Aldrich (www.Sigma-Aldrich.com). A 5-member set of DBS dose-response-curve materials, enriched with 0–15 µmol SUAC/L blood, was among the prepared PT pools. This set of materials was made from a single unit of hematocrit-adjusted non-enriched blood to ensure that the endogenous SUAC concentration was the same for all members of the set. The blood for all DBS PT pools was dispensed in 75 µL aliquots onto Whatman 903 (www.whatman.com/SpecimenCollectionDevices.aspx) or Ahlstrom 226 filter paper (www.perkinelmer.com/pages/060newbornscreening/default.xhtml). After overnight drying, the DBS cards were separated with sheets of weighing paper (www.fishersci.com) and packaged for storage at −20°C and controlled low humidity (<30%) in zip-closure liquid-tight specimen bags (com-pac.com) containing desiccant packets (polylam.com) and humidity indicator cards (www.desiccare.com) to await analytic characterization by NSQAP’s routinely used MS/MS method [8] and subsequent distribution to PT program participants.

For each SUAC PT event, identical 5-member panels of DBS specimens from characterized PT pools were packaged in zip-closure Mylar-foil bags (www.impakcorp.com/) containing fresh desiccant packets and distributed at ambient temperatures to all active PT program participants. Participants were instructed to report the type of method used for SUAC analyses, the SUAC concentration of each PT specimen, the cutoff value used to sort SUAC test results and each specimen’s presumptive clinical classification (within or outside normal limits).

2b. Quantitative data analyses

The arithmetic mean of all SUAC concentrations reported for each PT specimen was computed and values outside the 99% confidence interval around each mean were excluded before the quantitative-data analyses were begun. All reported cutoff values and clinical assessments were included in the analyses of those data.

The quantitative results for each PT specimen were sorted into four analytic method groups—derivatized MS/MS non-kit, non-derivatized MS/MS non-kit, non-derivatized MS/MS NeoBase™ kit and other. The method category “other” contained results from colorimetric, fluorometric, porphobilinogen inhibition, and ALA dehydratase-inhibition non-MS/MS methods, a non-kit LC-MS/MS method,and a derivatized MassChrom LC-MS/MS kit (www.chromsystems.com) method. Not all methods categorized as “other” were represented in every quarter.

Linear regression analyses were used to examine the comparability by method of reported SUAC concentrations in the DBS set of SUAC dose-response-curve materials. Bias plots of representative specimens from PT panels distributed in 2009, 2010, and 2011 were used to identify consistent patterns of method-related biases in SUAC quantitation over time and over the SUAC concentration range of the PT specimens. The bias plots were constructed to show, by laboratory and method, the difference (positive or negative) of the SUAC enrichment value subtracted from the reported value. Each plot also shows the mean participant bias (the mean of all participants’ assayed values minus the enrichment value) and the 95% confidence interval around the mean bias.

Participants were asked to report their cutoff value—the decision level for sorting test results reported as presumptive positive (outside limits) from results reported as negative (within limits)—used to classify the specimens in each PT event. All reported SUAC cutoff values were grouped to show, by PT event, the arithmetic mean cutoff value for each analytic method type, and within the two large method groups (derivatized non-kit and non-derivatized NeoBase™ kit MS/MS methods), the data were sorted to separate those from laboratories in the United States (domestic) and those from laboratories in other countries (foreign). The 2009 estimated births in each state within the United States [9] were used to classify states as large (>125,000 births per year) or small (<125,000 births per year), and domestic laboratories’ results were sorted into those reported by laboratories in large or small states.

2c. Qualitative data analyses

NSQAP’s assignment of final presumptive clinical assessments of its PT specimens is consistent with guidelines in the Clinical Laboratory Improvement Amendments regulations [10]. An NSQAP-classifiable specimen must have 80% or more agreement of classification among the United States’ domestic laboratories. For this report, a specimen was declared classifiable if it had 80% or more agreement of classification among all laboratories (domestic and foreign). All reported presumptive clinical assessments of each non-classifiable PT specimen were sorted by method group. Presumptive clinical assessments from the derivatized non-kit and non-derivatized NeoBase™ kit MS/MS method groups (which together accounted for about 80% of all assessments) were compared, and within each of these method groups, assessments from domestic and foreign NBS laboratories were compared.

2d. Comparison of SUAC concentrations in NBS samples from confirmed hepatorenal tyrosinemia patients

SUAC concentrations measured in the NBS samples of confirmed hepatorenal tyrosinemia patients were retrospectively collected from NBS laboratories in the United States. The anonymous patient sample data were sorted by the SUAC test method used to analyze the samples, and the ranges of SUAC concentrations found in the two method groups were compared.

3.RESULTS

3a. Quantitative results

Linear regression analyses (Figure 1) show, by analytic method, average reported SUAC concentrations versus enriched SUAC concentrations of a set of DBS dose-response materials distributed in Quarter 4 2010. The Y-intercepts derived from linear regression analyses provide one measure of the endogenous content of the blood matrix used to prepare the dose-response materials. Endogenous concentrations were also measured by analysis of the non-enriched member of the set of dose-response materials. For each method group, the endogenous SUAC concentrations derived from linear regression analysis and biochemical analysis were similar. Regression slopes derived from analyses of the SUAC dose-response materials ranged from 0.2 (NeoBase™ method) to 0.8 (non-derivatized MS/MS non-kit methods.)

Figure 1.

Linear regression analyses of results from a set of dried-blood-spot succinylacetone dose-response materials analyzed by Newborn Screening Quality Control Program participants in Quarter 4 2010.A

^A N-values indicate the number of laboratories represented in each method group.

The bias plots in Figure 2 show, by laboratory and method, the difference (positive or negative) of the SUAC enrichment value subtracted from the reported value. A reported value matching the enrichment value will fall on the zero-line of the plot. The data show a tightly clustered set of values with a negative bias for the NeoBase™ method. The bias values for SUAC have a wide scatter and a large difference among methods and users. Only a few SUAC participants showed good recoveries relative to the enrichment values. A marked difference was observed between quantitative results from the derivatized non-kit and NeoBase™ kit MS/MS methods. Reports from NeoBase™ kit users made up 28% of results reported for the first 2009 PT event and 48% of results reported for the last 2011 PT event.

Figure 2.

Bias plots of succinylacetone (SUAC) concentrations in dried-blood-spot proficiency testing specimens grouped by analytic method. Quarter 4 2009–Specimen 4934 (SUAC enrichment = 15 µmol/L blood) (A), Quarter 3 2010–Specimen 3033 (SUAC enrichment = 2.5 µmol/L blood) (B), Quarter 4 2011–Specimen 4131 (SUAC enrichment = 30 µmol/L blood) (C).

^A MS/MS = tandem mass spectrometry

Figure 3 shows average cutoff values reported for each PT event. Overall, the highest average cutoff values were from foreign laboratories that used derivatized MS/MS non-kit methods, and the lowest were from foreign laboratories that used NeoBase™ kits. Among users of derivatized MS/MS non-kit methods, average cutoff values from foreign laboratories were higher than those from domestic laboratories; among NeoBase™ kit users, average cutoffs from foreign laboratories were lower than those from domestic laboratories. Figure 3 depicts the average cutoff values of all domestic NeoBase™ kit users; however, the averages of cutoffs reported by large-state kit users were higher than the averages of cutoffs reported by small-state kit users. During 2011, the average quarterly cutoff values of the 10 large-state kit users ranged from 4.0–4.3 µmol SUAC/L blood, and those of the 3-to-7 small-state kit users ranged from 1.5–1.6 µmol SUAC/L blood.

Figure 3.

Average succinylacetone cutoff valuesA,B of domestic and foreign laboratories that used derivatized non-kit and underivatized PerkinElmer NeoBase™ kit tandem mass spectrometry (MS/MS) methods in 2009–2011.

• = Domestic non-kit users, = Foreign non-kit users,
• = Domestic NeoBase™ kit users, ♦ = Foreign NeoBase™ kit users

^A The decision level for sorting test results reported as presumptive positive (outside limits) from results reported as negative (within limits).

^B The number of reported cutoff values represented in each data point is indicated on the chart.

3b. Qualitative results

For the 2009–2011 PT surveys reported here, a total of 8 PT specimens, enriched with 2.5–15.0 µmol SUAC/L blood, did not meet our 80% criterion for consensus classification. For those 8 specimens, we compared the clinical classifications of results from derivatized non-kit and NeoBase™ kit MS/MS methods to determine whether the method-related differences in quantitation affected the presumptive clinical classifications of the specimens. Additionally, we sorted the classifications within each method group by laboratory type (domestic or foreign) to investigate the possibility that NBS practices affected clinical classifications (Table 1). Among domestic and foreign laboratories that used derivatized non-kit MS/MS methods, specimens enriched with 2.5–5.0 µmol SUAC/L blood were not classifiable, and all specimens with SUAC enrichments ≥10 µmol SUAC/L blood were classifiable as outside normal limits. Among NeoBase™ kit users, presumptive clinical classifications reported by domestic and foreign laboratories were concordant for only 1 of the 8 specimens (Table 1). The domestic kit-user group classified specimens enriched with 10 µmol SUAC/L blood as within normal limits and did not reach classification consensus for the specimen enriched with 15 µmol SUAC/L blood, whereas, the foreign kit-user group classified all specimens with SUAC enrichments of ≥10 µmol SUAC/L blood as outside normal limits. The foreign kit-users’ consensus classifications of results from specimens enriched with ≥10 µmol SUAC/L blood were concordant with the classifications derived from results of non-kit MS/MS methods. Overall, the SUAC enrichment range resulting in non-concordant clinical classifications was 2.5–5.0 µmol SUAC/L blood among laboratories that used derivatized non-kit MS/MS methods and 2.5–15.0 µmol SUAC/L blood among NeoBase™ kit users.

Table 1.

Presumptive clinical classifications of succinylacetone (SUAC) proficiency testing specimens that did not achieve overall clinical classification consensus^A

			Derivatized MS/MSB non-kit methods				NeoBase™ Kit method
SUAC			Domestic labs		Foreign labs		Domestic labs		Foreign labs
Enrichment (µmol/L blood)	Year & Quarter	Specimen Number	Reports WNLC/ONLD	Classifi- cation	Reports WNL/ONL	Classifi- cation	Reports WNL/ONL	Classifi- cation	Reports WNL/ONL	Classifi- cation
2.5	2010-3	3033	8|3	NCE	9|3	NC	11|10	WNL	6|2	NC
3	2009-3	3933	6|4	NC	9|5	NC	11|10	WNL	6|1	WNL
5	2010-1	1035	4|7	NC	7|8	NC	2|1	NC	1|7	ONL
5	2010-4	4033	6|5	NC	7|7	NC	12|10	WNL	4|5	NC
10	2010-3	3035	0|11	ONL	0|12	ONL	9|2	WNL	0|8	ONL
10	2010-4	4034	1|10	ONL	1|13	ONL	10|2	WNL	1|8	ONL
10	2011-3	3131	2|8	ONL	2|13	ONL	9|8	NC	2|12	ONL
15	2010-4	4035	0|11	ONL	1|13	ONL	5|7	NC	0|8	ONL

^AAgreement of classification by ≥80% of all participating laboratories is needed for consensus

^BTandem mass spectrometry

^CWNL = within normal limits (classification reported by ≥80% of laboratories in this group)

^DDNL = outside normal limits (classification reported by ≥80% of laboratories in this group)

^ENC = non-classifiable (<80% agreement of classification by laboratories in this group)

3c. Patient sample results

Several NBS laboratories in the United States have voluntarily sent NSQAP the anonymous SUAC concentrations found in the NBS samples of their confirmed hepatorenal tyrosinemia patients. These NBS results, summarized in Table 2, showed no overlap of the ranges of SUAC concentrations measured by derivatized non-kit and NeoBase™ kit MS/MS methods. The minimum value from the set of SUAC concentrations obtained by derivatized MS/MS non-kit methods was 15.0 µmol SUAC/L blood; the maximum value obtained by NeoBase™ kit was 14.8 µmol SUAC/L blood.

Table 2.

Succinylacetone (SUAC) concentrations in newborn screening samples of confirmed hepatorenal tyrosinemia patients identified by tandem mass spectrometry (MS/MS) methods

			SUAC concentrations (µmol/L blood)
MS/MS Method	Number of States	Number of Patients	Range of CutoffsA	MaxB	MinC	Mean	Median
Non-kitD	3	6	3.0–5 5	39	15	21.8	19
KitE	3	7	1.4–4.5	14.8	6.1	10.3	10.5

^ACutoff = decision level for sorting test results reported as presumptive positive (outside limits) from results reported as negative (within limits)

^BMax = maximum reported concentration

^CMin = minimum reported concentration

^DDerivatized non-kit MS/MS methods

^ENon-derivatized NeoBase™ kit method

4. DISCUSSION

From the first PT event of 2009 through the last PT event of 2011, the number of laboratories reporting SUAC PT results increased from 31 to 69. Between July 2009 and November 2011, the numbers of laboratories reporting results from derivatized MS/MS non-kit methods, non-derivatized MS/MS non-kit methods, and the group of methods categorized as “other” have fluctuated between 23–25, 1–5, and 3–6 per quarter respectively, whereas the number of laboratories using NeoBase™ kits has increased from 8 to 33.

The range of regression line slopes shown in Figure 1 (0.2 to 0.8) indicates method-specific differences in recoveries of SUAC from DBS PT specimens. Ideally, the slope should be 1.0. Generally, slope values substantially different from 1.0 indicate that a method has analytic bias. Because the endogenous SUAC concentration was the same for all pools within a dose-response set, it should not affect the regression line slopes.

The bias plots show tightly clustered NeoBase™ kit method results which are low-biased relative to both SUAC enrichment and the overall mean of all reported results. This data grouping, which was first observed during the SUAC pilot study [7], illustrates the harmonizing contributions of using a common test method calibrated with a single-source internal standard prepared according to a common protocol. The negative bias for results from the NeoBase™ kit method combined with increased use of that kit has resulted in increased differences between the enriched SUAC concentration and the mean participant bias.

The basis of each NBS program’s presumptive clinical classification decisions is related to its assigned cutoff value, which is derived from using the selected method to analyze a sizeable number of unaffected patient specimens. The reported screening data reflect analytic bias; therefore, laboratories that reported higher quantitative results were expected to use and report higher cutoff values. As expected, the average cutoff values calculated from derivatized non-kit MS/MS users’ cutoff values were higher than the averages calculated from foreign laboratories’ NeoBase™ kit cutoff values and were initially higher than those calculated from domestic laboratories’ NeoBase™ kit cutoff values (Figure 3). However, after the first 2 quarters of 2009, the average cutoff values for domestic kit users increased to levels comparable to those of domestic non-kit MS/MS methods users. This increase in domestic kit users’ average cutoffs was attributable to high cutoff values contributed by large-state laboratories.

Presumptive clinical classifications of results from PT specimens enriched with 10–15 µmol SUAC/L blood and analyzed by large-state NeoBase™ kit users were different from the classifications submitted by other domestic and foreign NSQAP participants. These discordant classifications were attributable to the large states’ higher cutoff values. Commonly, NBS laboratories set conservative initial cutoff values to minimize the risk of false-negative reports and will revise their initial cutoff values to less conservative levels as their NBS experience and data base justify. Thus, the higher cutoffs of large-state NeoBase™ kit users, relative to those of other NeoBase™ kit users, may reflect differences in the sizes of their respective data bases. Among domestic laboratories that used non-kit MS/MS NBS methods, the large-state cutoff value fell within the range of small-state cutoffs, and small- and large-state classifications of PT specimens enriched with 10–15 µmol SUAC/L blood were consistent with those submitted by foreign laboratories for these methods.

The reported patient-sample SUAC results are from hepatorenal tyrosinemia patients whose NBS tests were performed in the United States and include the recently published NBS SUAC concentrations of New York’s first confirmed hepatorenal tyrosinemia patients [11]. The range of SUAC concentrations found in the NBS samples of the confirmed tyrosinemia patients who were screened by derivatized non-kit MS/MS methods is higher than the range found in the NBS samples of those who were screened by the NeoBase™ kit method (Table 2). These results suggest that patient sample reference ranges may be test-method-specific because of analytic bias similar to that reported for PT specimens; however, the number of patient sample results shown here is small and not comprehensive. An ongoing compilation of MS/MS NBS results from laboratories worldwide has been undertaken by the MS/MS Collaborative Project in Region 4 of the US Regional Genetics and Newborn Screening Collaboration [12].

In summary, results from DBS PT specimens showed a marked difference between SUAC concentrations measured by derivatized non-kit and NeoBase™ kit MS/MS methods—the two NBS method types presently used by domestic NBS laboratories. Overall, quantitative results and cutoff values reported by the non-kit MS/MS methods users were higher than those reported by the kit users; however, most large-state kit users’ cutoff values were markedly higher than those of all other NeoBase™ kit users. All derivatized non-kit MS/MS methods users and foreign NeoBase™ kit users reached classification consensus of “outside normal limits” for DBS specimens enriched with 10–15 µmol SUAC/L blood, but domestic NeoBase™ kit users did not. The discordant presumptive clinical classifications of domestic kit users were attributable to the high cutoff values used by most large-state laboratories. The NBS SUAC concentrations of confirmed hepatorenal tyrosinemia patients’ samples illustrated test-method-related differences that were consistent with those observed for reported SUAC concentrations of NSQAP’s PT specimens.

Between January 2009 and October 2011, the number of participants reporting SUAC results for NSQAP’s PT events increased from 31 to 69. In this new and rapidly expanding area of NBS, programs initiating screening for hepatorenal tyrosinemia should be aware of the reported analytic biases among NBS SUAC tests and establish conservative initial cutoff values appropriate for their selected method and their newborn population tested. Building a reliable quality control (QC) system for monitoring the performance of the SUAC assay is critical for understanding and controlling the variables that may influence the outcomes of the screening tests. Ongoing internal and external evaluations of presumptive clinical assessment practices and performance, in conjunction with a robust analytic QC system, contribute to confidence in the SUAC assay and reduction of false-positive rates with minimized risk of false-negative reports. Through its two routine distribution components— QC materials for periodic use and quarterly PT—NSQAP enables programs to monitor the performance of their SUAC assays, compare their quantitative results with those of others who use the same and different methods, and receive assessments of their performance [13]. The cumulative PT results from NSQAP’s participating laboratories will contribute substantially toward building worldwide harmonization of SUAC NBS tests results. In the interim, screening is by definition [14] about sorting out apparently well persons (asymptomatic individuals) who probably have or will have a disease from those who probably do not or will not, and analytic biases among laboratories may be tolerated so long as the risk for misclassifications are negligible.

Abbreviations

SUAC

succinylacetone

MS/MS

tandem mass spectrometry

NBS

newborn screening

NSQAP

Newborn Screening Quality Assurance Program

CDC

Centers for Disease Control and Prevention

PT

proficiency testing

DBS

dried-blood spot

QC

quality control