Reliability and Validity of Genotyping Filaggrin null mutations

Atopic dermatitis (AD) is a common chronic waxing and waning disease ¹. Of those with AD and European or Asian ancestry, more than 20% will have a FLG loss of function mutation (FLG null) resulting in the absence or near absence of filaggrin protein in the stratum corneum ^2–5. It has been suggested that FLG null genetic testing might be a helpful clinical tool ^2;4. The FLG gene is difficult to sequence because of its large, redundant, and repetitive structure limiting the availability of specific primer binding sites for PCR amplification ^2;3;5. There are several techniques that are commonly used for genetic testing including the more time intensive TaqMan® allelic discrimination assays and high-throughput methods that utilize beadchip technology ^3–6. The goal of this study was to compare the reliability (the consistency of and ability to replicate a measurement) and validity (the ability of a measurement to identify the correct event) of these two techniques.

We obtained DNA from a cohort of more than 750 children ^4;7Using both methods, we genotyped samples for the four most prevalent FLG null mutations found in European populations:R501X; 2282del4; R2447X and S3247X ². Subjects were also categorized as: absence of a mutant allele, presence of one mutant allele, or presence of more than one mutant allele (i.e., null homozygotes or null compound heterozygotes (<0.1%)). Since one FLG null allele is sufficient to create a clinical phenotype, we also created a dichotomous variable for each mutation based on the presence or absence of a single FLG null allele ^2;4;8. The reliability was expressed as Kendall’s Tau-b and Cohen’s Kappa and the validity as sensitivity and specificity.

Reliability ranges are conventionally interpreted as: 0–0.20 as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1 as almost perfect. When comparing two laboratory genetic tests, the common expectation is that the reliability should be almost perfect or perfect. Acceptable diagnostic tests often have sensitivity and specificity greater than 90% when compared to a gold standard. Validity of less than 95% is likely not acceptable for comparing non-invasive laboratory based tests.

Our beadchip was established with assistance from Illumina using previously published data for all four mutations ^3;9. Genotyping assays were conducted using the GoldenGate platform, a Bead Array Scanner, and the Genome Studio Software (Illumina Inc, San Diego, CA). About 15% of the samples were genotyped twice using the beadchip. The results were identical for both runs (i.e., perfect test-retest reliability).

For our samples, custom-made TaqMan® assays (Applied Biosystems, Foster City CA) were designed using previously published protocols ^3;9. TaqMan® genotyping for these mutations is the current gold standard and the application of this methodology to identify individuals who carry FLG null mutations and hence no or minimal filaggrin protein in the stratum corneum has been demonstrated using skin biopsy and Raman spectroscopy studies ^2;3;8. Importantly, our TaqMan® assays have been used to screen more than 2000 cases of AD including verification by DNA sequencing ^3;5, In our study, about 15% of samples were genotyped twice using TaqMan® and the results were identical for both runs (i.e., perfect test-retest reliability).

Our a priori expectation was that there should be near perfect agreement between the two genotyping methods (inter-test reliability). However, this was not observed (Table 1). Pearson’s Chi squared tests indicated that statistically significant differences existed between the two techniques (p< 0.0001 for all mutations). The values for the two reliability measures, Cohen’s Kappa and Kendall’s tau-b are presented in Table 1. Reliability assessments were mostly “fair to moderate” and none were “almost perfect”. The sensitivity and specificity (assuming TaqMan® as the gold standard) ranged between 36% and 87% for sensitivity and between 84% and 100% for specificity. The positive predictive value (PPV) for the beadchip technique properly predicting the TaqMan® technique for the combined FLG null assessment was 53%. The results did not vary by race (results not shown).

Table 1.

Reliability and validity comparisons for FLG null mutations as assayed by TaqMan® and beadchip methods.

Gene Variant	Reliability		Validity
	Kendall’s tau-b	Cohen’s Kappa	Sensitivity	Specificity
R501X	0.7075	0.6029	92.3 %	94.1 %
2282del4	0.6090	0.5806	69.2	96.4
R2447X	0.1605	0.0925	46.7	90.0
S3247X	0.5615	0.5050	36.0	99.9
Any FLG null	0.5568	0.4866	89.4	83.9

Our study demonstrated that when the same genotyping method was repeated, the results were always identical (perfect test-retest reliability). However, when samples were assayed with both TaqMan® and beadchip technologies, the results were often different, with reliability values mostly suggesting only moderate agreement (inter-rater reliability). In addition, the validity measures for the beadchip method, assuming TaqMan® as the gold standard, were frequently poor. The PPV of the beadchip method was only 53%. In other words, only about half the time when the beadchip revealed a FLG null mutation, was a mutation truly present per TaqMan®.

A potential limitation of this study could be the laboratory’s performance in the use of the beadchip technology. This is unlikely. First, the laboratory that conducted these tests is a University-based core facility that was established to perform genetic sequencing and is familiar with the technology. In addition, repeated assays were identical. Second, the chip was created by Illumina using the same information used to create the TaqMan® assays. All SNPs on the chip passed Illumina quality assurance. Third, the same chip was used to determine ancestry using 120 marker SNPs. The beadchip assays predicted self-reported race with more than 90% concordance to self-reported race⁴.

Published results using beadchip technology have never fully captured the magnitude of association seen between the presence of FLG variants determined by TaqMan® and AD ^6;10. Based on our findings this may be related to differences in the genotyping methods. It is likely that we should have expected our results in that the goal of high throughput techniques is often to find genes of interest using common tag SNPs and this technique is not ideal to assay redundent areas of the genome. TaqMan® techniques are often used to find specific causal mutations and can be more precisely customized. In conclusion, our study demonstrates that the beadchip technology may not be appropriate for genotyping the FLG gene. This technology should not be used for clinical or research purposes until futures studies are conducted better to better understand these differences and experimental proof are obtained with studies of filaggrin protein in the stratum corneum to verify bead chip results.