Quality Control Project of NGS HLA Genotyping for the 17^th International HLA and Immunogenetics Workshop

Abstract

The 17^th International HLA and Immunogenetics Workshop (IHIW) organizers conducted a Pilot Study (PS) in which 13 laboratories (15 groups) participated to assess the performance of the various sequencing library preparation protocols, NGS platforms and software in use prior to the workshop. The organizers sent 50 cell lines to each of the 15 groups, scored the 15 independently generated sets of NGS HLA genotyping data, and generated “consensus” HLA genotypes for each of the 50 cell lines. Proficiency Testing (PT) was subsequently organized using four sets of 24 cell lines, selected from 48 of 50 PS cell lines, to validate the quality of NGS HLA typing data from the 34 participating IHIW laboratories. Completion of the PT program with a minimum score of 95% concordance at the HLA-A, HLA-B, HLA-C, HLA-DRB1 and HLA-DQB1 loci satisfied the requirements to submit NGS HLA typing data for the 17^th IHIW projects. Together, these PS and PT efforts constituted the 17^th IHIW Quality Control project. Overall PT concordance rates for HLA-A, HLA-B, HLA-C, HLA-DPA1, HLA-DPB1, HLA-DQA1, HLA-DQB1, HLA-DRB1, HLA-DRB3, HLA-DRB4 and HLA-DRB5 were 98.1%, 97.0% and 98.1%, 99.0%, 98.6%, 98.8%, 97.6%, 96.0%, 99.1%, 90.0% and 91.7%, respectively. Across all loci, the majority of the discordance was due to allele dropout. The high cost of NGS HLA genotyping per experiment likely prevented the retyping of initially failed HLA loci. Despite the high HLA genotype concordance rates of the software, there remains room for improvement in the assembly of more accurate consensus DNA sequences by NGS HLA genotyping software.

1. Introduction

In the first Workshop on Histocompatibility Testing in 1964, the tissue typing community met to share and evaluate cells, reagents and typing methods with the goals of understanding variation between different tests and identifying best practices for tissue typing efforts moving forward [1]. Over the last 50 years, the 15 subsequent International HLA and Immunogenetics Workshops (IHIW) have served as fora for the exchange of knowledge and experience, evaluating new methods, establishing technological standards and advancing ongoing collaborative projects [2–16]. From the 2^nd to the 13^th IHIW, participants performed parallel tests with official workshop reagents; the data generated with those reagents were submitted for central analysis. In order to obtain high quality, definitive results for each of these workshops, their organizers instituted quality control (QC) requirements for participating laboratories. These QC exercises included pre-testing of blind samples or inclusion of blind reagents. Only laboratories that met these requirements could submit data for central analysis.

When PCR-based molecular typing methods were first investigated in the11^th IHIW [11], only 189 HLA alleles were known [17]. By the 16^th IHIW [16], PCR and Sanger sequencing-based typing (SBT) methods had proliferated, and 7,527 HLA alleles were known (IPD-IMGT/HLA Database release version 3.8). However, variation in PCR-based typing methods and SBT methods has made it challenging to understanding how HLA allele data can best be applied for clinical and research ends. Since the 16^th IHIW, next-generation sequencing (NGS) based genotyping technology [18], which can potentially sequence entire HLA genes, has been seen as a means to address these challenges.

The 17^th IHIW was held in Northern California in the fall of 2017, and focused on the application of NGS for histocompatibility, immunogenetics and immunogenomics. A principal goal of the 17^th IHIW was to provide an opportunity to introduce NGS methods to participating laboratories, and for those laboratories to become proficient with and further refine their use of NGS technology. Here, we describe both a Pilot Study (PS) that evaluated available NGS DNA sequencing library protocols, sequencing platforms and genotyping software prior to the 17^th IHIW, and the 17^th IHIW Proficiency Testing (PT) program, which was applied to evaluate the NGS genotyping performance of each participating 17^th IHIW laboratory. These PS and PT efforts constituted the 17^th IHIW QC project. Though more than 50 years have passed, the aim in these efforts is largely the same as in the first Histocompatibility Workshop – identifying best practices for NGS HLA genotyping efforts in the 21^st Century.

2. Materials and Methods:

2.1. NGS HLA Reference Panel (RP):

An NGS HLA Reference Panel (RP), constructed from 50 cell lines collected in previous IHIWs [13] supplied by the Fred Hutchinson Cancer Research Center (FHCRC) (https://www.fredhutch.org/en/labs/clinical/projects/ihwg.html),was distributed blindly for the PS to an international collection of 13 laboratories applying different platforms and/or reagents for NGS HLA typing (Table 1). PS laboratory 13 used three different NGS HLA genotyping protocols and software. Overall, the PS included HLA genotyping data from 15 (= 12 + 3) independent experiments (15 groups). Cell lines were selected to represent a wide range of HLA allele groups and to include common and well documented (CWD) [19] and non-CWD alleles [20], null alleles, and the RP included cell lines that were homozygous for at least one locus. These RP cell lines had been typed previously by Sanger sequence Based Typing (SBT) [21], sequence-specific primers (SSP), sequence-specific oligonucleotide probe (SSO) [22], and serological and cellular methods for some but not all HLA genes.

Table 1.

NGS HLA Pilot Study (PS) and Proficiency Testing (PT) participating laboratories

	Laboratory	PS	PSN	PT	PTN
1	Department of Blood Group Serology and Transfusion Medicine, Medical University of Vienna, Austria	Y	1	N
2	Uppsala University, Uppsala, Sweden	Y	1	N
3	Georgetown University Medical Center, Washington DC, USA	Y	1	N
4	Bo Fu Rui (BFR) Transplant Diagnostics, Beijing, China	Y	1	N
5	Histocompatibility and Immunogenetics Laboratory, Nantes, France	Y	1	N
6	Royal Perth Hospital, Perth, Australia	Y	1	N
7	Transplantation and Immunology, Tuebingen, Germany	Y	1	N
8	Transplantation Immunology, Ulm, Germany	Y	1	N
9	GenDx, Utrecht, Netherlands	Y	1	N
10	University of North Carolina, Chapel Hill, NC, USA	Y	1	N
11	University of California, Los Angeles, Immunogenetics Center, Los Angeles, CA, USA	Y	1	Y	1
12	Anthony Nolan Research Institute and UCL Cancer Institute, Royal Free Campus, London, UK	Y	1	Y	1
13	Stanford Blood Center,Palo Alto, CA, USA	Y	3	Y	1
14	All India Institute of Medical Sciences, New Delhi, India	N		Y	1
15	Ente Ospedaliero Ospedali Galliera, Genoa, Italy	N		Y	1
16	Fondazione I.M.E. Istituto Mediterraneo Di Ematologia, Rome, Italy	N		Y	1
17	Alexandrovska Hospital, Sofia, Bulgaria	N		Y	1
18	Histocompatibility/Molecular Genetics, American Red Cross, Philadelphia, PA, USA	N		Y	1
19	Health Sciences Center, Kuwait University, Jabriya, Kuwait	N		Y	1
20	Hellenic Cord Blood Bank, Athens, Greece	N		Y	1
21	Baylor University Medical center, Dallas, TX, USA	N		Y	1
22	Hospital Albert Einstein, Sao Paulo, Brazil	N		Y	1
23	illumina, Inc., San Diego, CA, USA	N		Y	1
24	Johns Hopkins University School of Medicine, Baltimore, MD, USA	N		Y	1
25	Kashi Clinical Laboratories, Inc., Portland, OR, USA	N		Y	1
26	McGill University Health Centre, Montreal, QC, Canada	N		Y	1
27	New Zealand Blood Service	N		Y	1
28	University of Miami Miller School of Medicine, USA	N		Y	1
29	One Lambda,Thermo Fisher Scientific, Canoga Park, CA, USA	N		Y	1
30	Palacky University, Faculty of Medicine and Dentistry, Olomouc, Czech Republic	N		Y	1
31	PathWest, Fiona Stanley Hospital, Murdoch, WA, Australia	N		Y	1
32	Primer Centro Argentino de Immunogenetica (PRICAI), Fundacion Favaloro, CABA, Argentina	N		Y	1
33	Walter Reed Army Institute of Research, Silver Spring, MD, USA	N		Y	1
34	Centro de Diagonóstico Biomédico, Hospital Clínic de Barcelona, Barcelona, Spain	N		Y	1
35	University of Tokyo, Tokyo, Japan	N		Y	1
36	The University of Chicago Medicine, Chicago, IL, USA	N		Y	1
37	Tokai University School of Medicine, Kanagawa, Japan	N		Y	1
38	Australian Red Cross Blood Services, Australia	N		Y	1
39	Cambridge University Hospitals NHS Foundation Trust, Cambridge, UK	N		Y	1
40	University Medical Center Utrecht, Netherlands	N		Y	1
41	City of Hope National Medical Center, Duarte, CA, USA	N		Y	1
42	University of California San Diego, La Jolla, CA, USA	N		Y	1
43	National H&I Service Development Laboratory NHS Blood and Transplant, London, UK	N		Y	2
44	Rogachev Federal Research Centre of Pediatric Hematology,Oncology and Immunology, Moscow, Russian Federation	N		Y	2

Table 1 identifies the laboratories that participated in the Pilot Study (PS), Proficiency Testing (PT) program or both. In the PS and PT columns, “Y” indicates laboratories that participated in the project, and “N” identifies the laboratories did not participate in the project. The PSN column indicates the number of NGS HLA genotyping protocols applied for PS genotyping in each laboratory. The PTN column indicates the number of NGS HLA genotyping protocols applied for PT genotyping. Three Laboratories reported results using two or more methods (See rows 13, 43 and 44).

The participating PS laboratories performed HLA genotyping for Class I (HLA-A, -B, -C) and almost all Class II (HLA-DPA1, -DPB1, -DQA1, -DQB1, -DRB1, -DRB3, -DRB4, -DRB5) genes using various commercially available or in house protocols, NGS platforms and HLA typing software. HLA allele calls were submitted in a spreadsheet format for the PS. The consensus HLA genotypes were generated by manual inspection of the results from the 15 independently generated HLA genotyping datasets, and were subsequently used as reference HLA genotypes for the subsequent PT program (See section 2.3). Of 886 alleles from 50 PS cell lines, one laboratory cloned 70 HLA alleles from 39 cell lines in E. coli to isolate the individual alleles, determined the cloned DNA sequences using the Sanger sequencing, and generated HLA allele calls (Barsakis et al., manuscript in preparation). When cloned HLA genotypes were available, consensus HLA genotypes were verified on the basis of the cloned allele sequences. All PS consensus genotypes were imported into the IHIW database [23].

2.2. NGS HLA Sequencing:

The focus of the workshop was the use of NGS HLA genotyping methods. For this reason, SBT, SSO and SSP typing results were not accepted if participants could not perform NGS HLA genotyping. MiSeq (Illumina), PacBio RS II (Pacific Biosciences) and Ion Torrent PGM sequencing instruments were used for both PS and PT genotyping. Some groups used the GS Jr. (Roche 454) for the PS, and the Ion Torrent S5 (Thermo Fisher) for PT. Table 2 shows the software used for PT NGS HLA genotyping.

Table 2.

Software used to perform PT NGS HLA genotyping

Software	Manufacturer	(n) Laboratories
Assign TruSight HLA	Illumina	8
HLA Twin	Omixon	5
MIA FORA	Immucor	3
NGSengine	GenDx	8
Omixon Target	Omixon	1
SeaBass (in house)	TOKAI University	1
SMRT Analysis/(in house)	Pacific Biosciences	1
TypeStream Visual	Thermo Fisher Scientific	8

2.3. Proficiency Testing Program and Proficiency Testing Panels:

To ensure the high quality of 17^th IHIW genotyping data, participants were required to submit NGS HLA genotyping results performed on a PT panel. Forty-eight RP cell lines were selected to construct four PT panels (PT1 – PT4), each of which consisted of 24 different RP cell lines. Supplemental Table 1 identifies the cell lines included in each PT. All participating laboratories submitting NGS HLA data to the 17^th IHIW were required to type one PT panel. The cell panels were shipped to the laboratories from the IHWG Cell and DNA Bank, along with recommended handling instructions for genomic DNA. Each cell line DNA was labeled with a coded 17^th IHIW sample ID (Supplemental Table 1), and shipped at a 100 ng/μl concentration in a total volume of 20μl per tube (2μg of DNA). The individual PT evaluation results for each laboratory are confidential; PT results presented here have been intentionally disassociated from the associated PT laboratory’s identity.

2.4. Sequencing data standardization and validation:

NGS HLA genotypes for all the NGS HLA related projects, including the PT project, were validated and collected using the 17^th IHIW database [23]. HLA genotypes were imported in Genotype List (GL) String format [24]. The associated meta-data (e.g., consensus sequences, the reagents, sequencing instrument(s) and software used, and pertinent IPD-IMGT/HLA database version) were also imported into the 17^th IHIW database. To avoid allele name discrepancies arising from the use of different IPD-IMGT/HLA Database versions by different laboratories, a “LiftOver” process converted all HLA allele names to IPD-IMGT/HLA Database version 3.25.0 for data analysis, while the submitted HLA genotypes were maintained in the database [23]. The submitted genotypes were occasionally reviewed at the request of participants in response to their PT evaluations (section 2.5); in these cases, discordant scores resulted from allele name differences between IPD-IMGT/HLA Database versions. For example, HLA-DPA1*02:07 appeared in IPD-IMGT/HLA Database version 3.26.0, but did not exist in version 3.25.0. The LiftOver logic in the 17th IHIW database system converted HLA-DPA1*02:07 to HLA-DPA1*02:01:01:01 [23], while the PS consensus genotype for this allele, typed under IPD-IMGT/HLA Database version 3.25.0, was HLA-DPA1*02:02:01.

2.5. PT Results Evaluation:

Each cell line genomic DNA included in the RP or a PT panel was assigned a unique IHIW sample ID (Supplemental Table 1). The data uploaded for the 24 PT cell lines included the corresponding IHIW sample ID, allowing the PT project leaders to compare the results submitted for each cell line to the PS consensus genotyping. This evaluation was performed using the HLAGenotypeEvaluator software (https://github.com/IHIW/hlaGenotypeEvaluator), which was developed using the Java Programming Language at the Stanford Blood Center. HLAGenotypeEvaluator assigned a score for each allele tested (Table 3A). The PT evaluation results consisted of a column for each allele and locus with 3 rows per sample. The first row represents the HLA genotypes uploaded by participants. The second row shows PS consensus HLA genotypes. The third row contains the score for the comparison between the submitted PT genotypes and the PS consensus genotypes for each locus. See example in Supplemental Table 2.

Table 3A:

HLAGenotypeEvaluator scoring

Score	Category	HLA type	Description	Overall Analysis
Identical	Result	A*68:01:02:01	HLA genotype s are exactly identical	Identical or Concord ant
	Ref	A*68:01:02:01
Concordant	Result	DPA1*01:03:01:01	Both result and reference are concorda nt at least by two field assessm ent
	Ref	DPA1*01:03:01:02
AmbResultConcordant	Result	DQB1*05:03:01:01/DQB1*05:03:01:02	Ambiguit y reported in the result but not in the reference	Ambiguo us result
	Ref	DQB1*05:03:01:01
AmbRefConcordant	Result	DQA1*05:05:01:05	Ambiguity Reported in the reference but not in the result
	Ref	DQA1*05:05:01:01/DQA1*05:05:01:02
AmbRefAmbResultConcorda nt	Result	DRB1*07:01:01:02/DRB1*07:01:01:01/DRB1 *07:01:01:03	Ambiguity reported in both reference and result
	Ref	DRB1*07:01:01:01/DRB1*07:01:01:02
UnresolvedNullAmbResultCo ncordant	Result	B*15:01:01:01/B*15:01:14/B*15:26N	Concord ant, but null allele was found in the result ambiguity string	Unresolv ed Null
	Ref	B*15:26N
Discordant	Result	B*57:12	Indicates both alleles do not match by two field assessment	Discordant
	Ref	B*57:02:01

The “Score” column shows the terms assigned by the HLAGenotypeEvaluator software. The “Category” column indicates “Result” or “Ref (Reference)” for the “HLA type” column. The pre-fix “HLA-” was omitted in the “HLA type” column. Each “Result” row shows HLA allele call submitted by a project laboratory. Each “Ref” row presents the pertinent reference HLA allele. For example, 1) Identical: the result HLA allele (HLA-A*68:01:02:01) and reference HLA allele (HLA-A*68:01:02:01) are identical. 2) Concordant: the result (HLA-DPA1*01:03:01:01) is unambiguous and the reference (HLA-DPA1*01:03:01:02) is unambiguous; these alleles are concordant by two-field assessment (HLA-DPA1*01:03). The “Overall Analysis” column shows how the overall analysis was applied for each laboratory.

2.5.1. Identical versus Concordant:

The scoring of submitted PT genotypes as compared to PS consensus genotypes was applied to all fourth-field allele names. For example, if the PS consensus genotype was HLA-B*15:04:01:01 and a PT result was HLA-B*15:04:01, HLAGenotypeEvaluator assigned a “Concordant” score, as only three fields were reported by the participant, while the PS genotype included four fields (Table 3A). This type of discrepancy occurred for the data that was submitted before the organizers implemented strict allele name rules in the database [23]. Under the strict allele name rules, HLA-B*15:04:01 would not be accepted in the 17^th IHIW database if this allele was typed under IPD-IMGT/HLA Database version 3.25.0, because it is not an official 3.25.0 allele name. However, if HLA-B*15:04:01 were typed using IPD-IMGT/HLA Database version 3.24.0, the 17^th IHIW database system would have converted HLA-B*15:04:01 to HLA-B*15:04:01:01 using the database’s LiftOver system [23]. After the strict allele name rules were implemented to accept only HLA allele names with IPD-IMGT/HLA Database version 3.25.0, “Concordant” scores occurred in response to legitimate third- or fourth-field differences (e.g., HLA-C*03:04:01:01 vs. HLA-C*03:04:01:02) (Table 3A).

2.5.2. Concordant Ambiguities:

Ambiguities might have been reported in the submitted PT genotypes or the PS consensus genotypes. For example, if a PT genotype includes HLA-B*56:01:01:03, but the PS consensus genotype includes HLA-B*56:01:01:02/HLA-B*56:01:01:03, HLAGenotypeEvaluator assigned an “AmbRefConcordant” score. Conversely, if the ambiguity was reported in a PT genotype, HLAGenotypeEvaluator assigned an “AmbResultConcordant” score. If the ambiguity was reported in both PT and PS genotypes, the HLAGenotypeEvaluator assigned an “AmbRefAmbResultConcordant” score.

2.5.3. Null Ambiguities

The resolution of ambiguities containing null (non-expressed) alleles is clinically relevant. This prompted the 17^th IHIW workshop organizers to require that the ambiguities containing null alleles be excluded from Concordant Ambiguities. A separate score, “UnresolvedNullAmbResultConcordant”, was assigned to the results submitted without null ambiguities resolution (Table 3A and3B). Participants were required to resolve null ambiguities for the submission of subsequent experimental workshop data.

Table 3B:

Unresolved null allele ambiguity

Null allele ambiguity	Difference	Description
HLA-A*68:02:01:01/HLA-A*68:18N	Exon 2	20 bp insertion between codons 47 and 48
HLA-A*31:14N/HLA-A*31:01:02:01	Exon 4	Insertion of a “C” between codons 185 and 186
HLA-B*15:01:01:01/HLA-B*15:01:14/HLA-B*15:26N	Exon 3	Codon 99 TAC (Tyr) -> TAA (Stop)
HLA-B*44:02:01:03/HLA-B*44:19N	Exon 1	Deletion of a “G” in codon −23
HLA-DRB4*01:03:01:01/HLA-DRB4*01:03:01:02N/HLA-DRB4*01:03:01:03	Intron 1	Single nucleotide change G->A at the 3’ end of intron 1
HLA-DRB5*01:02/HLA-DRB5*01:10N	Exon 2	2bp deletion in codon 80

Observed null ambiguities are listed in column “Null allele ambiguity”. The null ambiguities likely result from shallow DNA sequence coverage (below the software threshold) for the specific exon or intron region shown in column “Difference”. The “Description” column describes the sequence change that results in the null allele.

2.6. Feedback to the participating laboratories

The PT evaluations were analyzed and reviewed by the organizers. The PT scores together with comments and recommendations were returned to the participating laboratories. An example of PT scores is provided as Supplemental Table 2.

3. Results:

Forty-four laboratories participated in the PS and/or PT programs (Table 1). Laboratory 13 applied three different NGS HLA genotyping protocols and software for the PS. The 17^th IHIW organizers generated PS consensus HLA genotypes from 15 independent NGS HLA genotyping experiments. Thirty-four laboratories submitted PT results. Of these, 32 laboratories submitted one set of HLA genotyping data. However, laboratories 43 and 44 submitted two independent sets of PT genotyping data generated using two different protocols and software (Table 1). Together the 17^th IHIW organizers collected 36 independent NGS PT HLA genotyping reports from 34 laboratories. Results from one laboratory were excluded from the final evaluation due to a sample mix up during testing by that laboratory. Results from 35 NGS HLA genotyping reports were included in the final evaluation (Reports 1 – 35 in Table 4). The report numbers assigned in Table 4 do not correspond to the author affiliations, or to the laboratory numbering in Table 1. Two laboratories that performed testing on subsets of PT cell lines were included and scored in the same way as the laboratories that requested full PT panels (Laboratories 34 and 35 in Table 4).

Table 4.

Scoring results analysis per laboratory

Report	PAN EL	(n) alleles per panel	TOTAL alleles tested (n)	Identical/ Concorda nt	Ambiguous Concordant (result and/or reference)	Combin ed: Identical concord ant Ambiguo us	Unresolv ed Null	Discord ant
1	PT1	423	377	78.2%	16.2%	94.4%	0.0%	5.6%
2			409	83.6%	13.4%	97.1%	0.0%	2.9%
3			328	77.1%	16.5%	93.6%	0.0%	6.4%
4			327	86.9%	13.1%	100.0%	0.0%	0.0%
5			288	87.2%	10.8%	97.9%	0.0%	2.1%
6			423	83.9%	14.9%	98.8%	0.2%	0.9%
7			423	85.1%	12.8%	97.9%	0.0%	2.1%
8			423	85.6%	14.4%	100.0%	0.0%	0.0%
9			327	87.5%	11.3%	98.8%	0.0%	1.2%
10			423	80.9%	18.9%	99.8%	0.2%	0.0%
11			423	90.5%	9.0%	99.5%	0.0%	0.5%
12			423	89.4%	8.5%	97.9%	0.0%	2.1%
13			423	90.3%	9.5%	99.8%	0.0%	0.2%
14	PT2	406	314	86.3%	13.7%	100.0%	0.0%	0.0%
15			368	75.0%	24.2%	99.2%	0.3%	0.5%
16			230	87.0%	11.3%	98.3%	0.0%	1.7%
17			406	90.1%	8.6%	98.8%	0.0%	1.2%
18			406	77.3%	21.9%	99.3%	0.0%	0.7%
19			314	90.4%	10.2%	100.6%	0.0%	0.0%
20	PT3	408	408	83.3%	13.0%	96.3%	0.0%	3.7%
21			408	89.2%	10.5%	99.8%	0.0%	0.2%
22			372	79.3%	13.2%	92.5%	0.0%	7.5%
23			230	81.7%	14.8%	96.5%	0.9%	2.6%
24	PT4	392	392	90.6%	9.4%	100.0%	0.0%	0.0%
25			299	60.2%	29.1%	89.3%	4.7%	6.0%
26			392	90.3%	8.9%	99.2%	0.0%	0.8%
27			383	87.5%	9.1%	96.6%	0.0%	3.4%
28			359	64.6%	13.1%	77.7%	1.1%	21.2%
29			352	79.5%	20.2%	99.7%	0.0%	0.3%
30			220	80.0%	15.5%	95.5%	0.0%	4.5%
31			392	88.8%	11.2%	100.0%	0.0%	0.0%
32			392	88.5%	11.0%	99.5%	0.0%	0.5%
33			264	92.0%	8.0%	100.0%	0.0%	0.0%
34	Randomly selected cells		49	65.3%	34.7%	100.0%	0.0%	0.0%
35			137	88.3%	11.7%	100.0%	0.0%	0.0%
ALL			12104	84.1%	13.5%	97.5%	0.2%	2.3%

Four PT Panels (PT1 – PT4) were prepared and used for PT. One panel was distributed to each of the participating laboratories. Laboratories #34 and #35 tested only selected cells. Results scored as Identical/Concordant or Ambiguous Concordant were given a “passing” score. The Lab number in column one does not correspond to the order of lab affiliations in Table 1.

3.1. Results per Laboratory:

The results were evaluated on the basis of the HLAGenotypeEvaluator score (Table 3A) and by compiling the results for each of the cell lines tested at each locus for each participating laboratory. Two scores (one per allele) were recorded per locus (except for HLA-DRB3, HLA-DRB4 and HLA-DRB5). The results were reported as “Identical/Concordant” in Table 4 by combining the “Identical” and “Concordance” scores from Table 3A. Results were reported as Ambiguous Concordant in Table 4 by combining all ambiguous scores except for those that contained null alleles from Table 3A and3B.

We combined “Identical”, “Concordant” and “Ambiguous Concordant” together as “Combined” concordant (Table 4). Eighteen laboratories reported greater than 99% combined concordance (Table 4). Twelve laboratories reported combined concordance between 95% and 99% (Table 4). Five laboratories reported combined concordance below 95%. Two laboratories performed the testing on randomly selected RP cell lines (instead of a designated panel); their results were not included in the “per panel analysis”. Overall combined concordance for Class I was 97.7% and 97.4% for Class II. The ambiguous concordant rates varied widely across laboratories (8% to 35%). The ambiguities are NGS library preparation protocol dependent: most derived from the application of different sets of PCR primers. Observed null ambiguities are listed in Table 3B. The null ambiguities result from shallow DNA sequence coverage (below the software threshold) for the specific exon or intron region shown in Table 3B. The discordances summarized in Table 4 likely resulted from combinations of PCR failure for the target loci, shallow DNA sequence coverage of some exons and allelic sequence imbalance.

3.2. Overall HLA Results:

The combined percent occurrence of each score was calculated for each Class I and Class II locus for all laboratories (Table 5 and Table 6). Overall average discordance was 2. 5% [2.2% (4541 out of 4646 total alleles) for Class I, and 2.6% (7266 out of 7460 total alleles) for Class II]. We compared concordance versus discordance for each locus. The highest incidence of discordant results (when Discordant and UnresolvedNull ambiguities are combined) was observed in HLA-DRB4 (10.0%) followed by HLA-DRB5 (8.3%). Most discordant PT genotypes resulted from either allele dropout or incorrect calls.

Table 5.

HLA Class I scoring analysis

SCORE	HLA-A	HLA-B	HLA-C
Identical	87.5%	86.7%	83.7%
Concordant	7.4%	5.2%	11.2%
AmbResultConcordant	3.2%	2.6%	3.2%
AmbRefConcordant	0.1%	0.3%	0.0%
AmbRefAmbResultConcordant	0.0%	2.2%	0.0%
UnresolvedNullAmbResultConcordant	0.2%	0.2%	0.0%
Discordant	1.7%	2.8%	1.9%

Table 6.

HLA Class II scoring analysis

SCORE	DRB1	DRB3	DRB4	DRB5	DQA1	DQB1
Identical	65.8%	69.5%	65.3%	91.7%	50.4%	72.7%
Concordant	7.6%	17.7%	0.5%	0.0%	27.3%	6.2%
AmbResultConcordant	11.1%	11.9%	0.5%	0.0%	13.3%	16.4%
AmbRefConcordant	5.3%	0.0%	21.7%	0.0%	6.1%	1.9%
AmbRefAmbResultConcordant	6.1%	0.0%	1.9%	0.0%	1.6%	0.4%
UnresolvedNullAmbResultConcordant	0.0%	0.0%	4.1%	0.7%	0.0%	0.0%
Discordant	4.0%	0.9%	6.0%	7.6%	1.2%	2.4%

Pre-fix HLA-was removed from each locus name.

3.3. Not Reported:

Each laboratory was required to perform NGS HLA typing on a PT panel. Completion of the PT program with a minimum score of 95% concordance at the HLA-A, HLA-B, HLA-C, HLA-DRB1 and HLA-DQB1 loci satisfied the requirements to submit NGS HLA typing data for the 17^th IHIW projects. PT cell lines that were not reported for HLA-A, HLA-B, HLA-C, HLA-DRB1 and HLA-DQB1 were scored as “Discordant”. Therefore, “Not Reported” indicates 0% for these 5 loci in Table 7. PT cell lines that were not tested or for which results were not uploaded for HLA-DPA1, HLA-DPB1, HLA-DQA1, HLA-DRB3, HLA-DRB4 and HLA-DRB5 were scored as “Not Reported”. If the participating laboratory did not submit results for a locus, that locus was scored as “Not Reported” (Table 7).

Table 7.

“Not Reported” results for Class I and Class II

Locus	Not Reported
HLA-A	0%
HLA-B	0%
HLA-C	0%
HLA-DPA1	33.2%
HLA-DPB1	11.1%
HLA-DQA1	28.0%
HLA-DQB1	0%
HLA-DRB1	0%
HLA-DRB3	9.8%
HLA-DRB4	7.0%
HLA-DRB5	3.2%

Twelve laboratories did not report results for HLA-DPA1, and 10 did not report results for HLA-DQA1. These laboratories did not sequence these loci. Eight laboratories did not report any results for HLA-DRB3, HLA-DRB4 and HLA-DRB5.

3.4. PT consensus genotypes:

After completion of scoring PT results from each laboratory against the PS consensus genotypes, we revised the consensus genotypes by generating PT consensus genotypes. Consensus DNA sequences were also analyzed to verify the accuracy of the updated consensus genotypes. The differences identified between PS and PT consensus genotypes were carefully reviewed and evaluated. For example, HLA-DQA1*03:03:01:01 allele calls for H000055A and H0000567 were updated to HLA-DQA1*03:03:01:03. This discrepancy occurred, because HLA-DQA1*03:03:01:03 was not included in the IPD-IMGT/HLA Database at the time of the PS, but was the best match in the IPD-IMGT/HLA 3.25.0 Database release version 3.25.0. This was also confirmed by the consensus sequence analyses. Supplemental Table 3 shows results of the PT consensus NGS HLA genotypes of all 48 RP cell lines. Bolded types represent cloned HLA genotypes.

4. Discussion

The 17^th IHIW QC project was conducted in four stages: 1) the PS using RP cell lines; 2) generation of consensus HLA genotypes from the PS results; 3) scoring PT results from each laboratory against the PS consensus HLA genotypes; and 4) generating consensus HLA genotypes from PT.

PT scores generated using HLAGenotypeEvaluator were analyzed to evaluate each laboratory’s performance as well as the frequency of each score per locus.

The overall “Ambiguous Concordant” results ranged from 8% to 35% (Table 4). We have identified several ambiguities that are unresolvable using the current NGS technologies and HLA genotyping software (manuscript in preparation). It appeared to be difficult to resolve some allele ambiguities even though the polymorphic sequence position was sequenced, particularly if the polymorphic sites are located in a homopolymer nucleotide sequence or short tandem repeat (STR). For example, a frequently observed allele ambiguity was HLA-DRB1*15:01:01:01/HLA-DRB1*15:01:01:02/HLA-DRB1*15:01:01:03. These alleles differ in STR length in intron 2. We suspect that this is most likely due to DNA polymerase slippage occurring during the initial PCR, making it difficult to accurately determine the STR copy number using current NGS technologies and HLA genotyping software (Rozemuller et al., manuscript submitted). In addition to these technical limitations, we also found that “Ambiguous Concordant” results may arise from the application of different HLA genotyping protocols and software. Understandably, different HLA genotyping protocols use different PCR primer annealing sites, resulting in the sequencing of slightly different regions of genes. These unresolvable ambiguities were the most common “Ambiguous Concordant” results.

We identified some genotype ambiguities that resulted from our LiftOver process when laboratories submitted HLA genotyping data generated under an IPD-IMGT/HLA Database version other than 3.25.0. For example, some laboratories reported HLA-A*02:11:01+HLA-A*68:01:02:01|HLA-A*02:69+HLA-A*68:01:01:01, where the PS consensus genotype for this cell line is HLA-A*02:11:01+HLA-A*68:01:02:01. One laboratory using IPD-IMGT/HLA Database version 3.28.0 reported the HLA genotype HLA-A*02:11:01+HLA-A*68:01:02:01|HLA-A*02:69+HLA-A*68:164. Our LiftOver process converted HLA-A*68:164 to HLA-A*68:01:01:01 because HLA-A*68:164 did not exist in IPD-IMGT/HLA Database version 3.25.0. This is why the 17^th IHIW organizers stressed the importance of generating HLA genotypes using IPD-IMGT/HLA Database version 3.25.0; it is difficult to compare HLA genotypes generated using different IPD-IMGT/HLA database versions even when attempts to standardize HLA genotypes data are made using a LiftOver process. The second example of genotype ambiguity that we observed was HLA-A*66:01:01+HLA-A*69:01|HLA-A*66:12+HLA-A*69:02. The PS consensus genotype for this cell line is HLA-A*66:01:01+HLA-A*69:01. This genotype ambiguity was most likely caused by the failure to phase exon2 and exon3 sequences. It was not evident whether this was a technical limitation of the NGS platform and software used or simply a poor sequencing result.

Null alleles were incorrectly identified at rates of 0.2% for HLA-A, 0.2% for HLA-B, 4.1% for HLA-DRB4 and 0.7% for HLA-DRB5 (Tables 5 and 6). The ambiguities containing null alleles observed for these loci were listed in Table 3B. For example, ambiguity HLA-A*31:14N/HLA-A*31:01:02:01+HLA-A*31:14N/HLA-A*31:01:02:01 was caused by poor exon4 sequence coverage. The HLA-B*15:01:01:01/HLA-B*15:01:14/HLA-B*15:26N ambiguity was caused by poor sequence coverage of HLA-B exon3. The HLA-DRB4*01:03:01:01/HLA-DRB4*01:03:01:02N/HLA-DRB4*01:03:01:03 ambiguity was caused by no sequence coverage or sequence coverage lower than the software’s detectable threshold across intron 1 and the intron1/exon2 boundary. The polymorphic site resulting in the HLA-DRB4*01:03:01:02N allele (G->A) is located at the end of intron1, and causes alternative splicing [25]. One of the difficulties in excluding HLA-DRB4*01:03:01:02N is that the nucleotide sequence of intron1 for this allele is not included in IPD-IMGT/HLA Database version 3.25.0. If the HLA genotyping software did not distinguish the expressed alleles from non-expressed alleles, the participating laboratory was required to review of the aligned nucleotide sequences and manually edit the genotype call. Some laboratories also reported heterozygous genotype HLA-DRB4*01:01:01:01+HLA-DRB4*01:03:01:01/HLA-DRB4*01:03:01:03 as homozygous (HLA-DRB4*01:03:01:01/HLA-DRB4*01:03:01:03), excluding the HLA-DRB4*01:01:01:01 allele. This was a relatively common discordant genotype.

We also performed analyses per PT panel, combining results for all laboratories that typed each panel; PT4 had the highest incidence of combined null ambiguities and discordance, because two laboratories had high rates: 10.7% and 22.3%, respectively (Table 4). The high incidences of allele dropouts resulted in this high discordant rate, because the 17^th IHIW organizers treated a “Not Reported” result as “Discordant” for HLA-A, HLA-B, HLA-C, HLA-DRB1 and HLA-DQB1.

A minimum score of 95% concordance between the PT results and PS consensus genotypes at the HLA-A, -B, -C, -DRB1 and -DQB1 loci was required for participants to submit NGS HLA typing data for 17^th IHIW projects. Throughout the PT project evaluation, it had become apparent that almost all the laboratories that participated in the PT project were sufficiently proficient to participate in workshop projects. Based on our observations, we recommend the following guidelines to improve concordance when performing NGS HLA typing:

1. It is very important to ensure the quality of the initial PCR step. Poor DNA quality often results in poor PCR performance, which directly affects HLA genotype quality, and in some cases results in complete HLA genotype dropout. For the PS and PT projects, we eliminated the issue of DNA quality as a factor contributing to HLA typing error by using a single cell bank to distribute standardized, high-quality DNA to each participating laboratory.

2. For PT scoring, the 17^th IHIW organizers decided to categorize “likely Dropout” as “Not reported” for the HLA-DPA1, HLA-DPB1, HLA-DQA1, HLA-DRB3, HLA-DRB4 and HLA-DRB5 loci, because it was difficult to determine if a missing genotyping result was intentionally not typed or represented a typing failure. There were many instances of potentially “likely Dropout” instances, where the participants may have tried to type a locus, but failed. If we had instead categorized “likely Dropout” as “Discordant”, then there would be many more discordant results. Future such PT and genotyping evaluation efforts should include a clear means for participants to distinguish intentionally untyped cases from cases of typing failure.

3. During the evaluation of NGS QC we occasionally observed unexpected results that were likely due to sample mix-ups rather than technical performance. We suggest that each NGS run includes QC measures to ensure that each sample’s identity and position are appropriately tracked. The PT panel cell lines that we established during this project can be a convenient resource to be used for quality control for the future NGS HLA genotyping.

4. The HLA community needs to be aware that any NGS HLA genotyping system will sometimes report an incorrect HLA allele assignment. It is critically important to review HLA genotypes prior to finalizing each HLA report. It may not be straightforward to capture such errors and to correct the HLA allele assignment in the NGS HLA genotyping software. Currently, some NGS HLA genotyping software may not allow HLA allele assignment correction, and the IHIW participants might have been reporting what was automatically reported on the software.

5. In some cases, it may be feasible to identify HLA allele dropouts if HLA haplotypes are reviewed. For example, one laboratory reported HLA-DRB1*11:01:01:01+HLA-DRB1*13:02:01 and HLA-DQB1*03:01:01:03+HLA-DQB1*03:01:01:03 for a cell line. In general HLA-DRB1*13:02~HLA-DQB1*03:01g haplotype is rare [26], indicating a potential HLA-DQB1 allelic dropout. This HLA-DQB1 allele dropout could have been identified if HLA-DRB1~HLA-DQB1 haplotypes had been reviewed.

We noted that no commercially available HLA NGS genotyping system provides a way to re-genotype individual loci for specific samples, or genotype all loci for only a single sample, because NGS HLA genotyping systems are designed to operate at “economies of scale”. In case where HLA genotyping failed for some loci, laboratories had to either repeat the entire experiment, or include the samples for which some loci had failed in their next NGS HLA genotyping experiment. It is likely that the high cost of repeating experiments for failed samples was the major obstacle for participating laboratories to achieve 100% concordance, because the 17^th IHIW organizer did not accept HLA genotype data generated using alternative methods, e.g., SSO, to identify the missing alleles.

Multiplatform comparisons of NGS HLA genotyping results are also limited in that raw sequence data generated using one vendor’s NGS protocol can often be only processed by the corresponding NGS genotyping software. We found that it was nearly impossible to re-generate consensus sequences from fastq data generated by one NGS vendor’s platform using a different vendor’s software. We generated consensus of consensus sequences for each allele of each cell, and assigned the newly generated consensus sequences to the reference sequences in IPD-IMGT/HLA Database version 3.25.0 to assign HLA alleles, and to verify the accuracy of the updated consensus genotypes from some PT results. During the process of generating consensus of consensus sequences, we found much room for improvement remains in assembling more accurate consensus DNA sequences, especially for the class II genes.

5. Conclusion:

The consensus HLA genotypes of the 48 cell lines from the complete NGS sequence of HLA genes performed during the 17^th IHIW is now available in Supplemental Table 3, and these cell lines are available from the IHWG Cell and DNA Bank repository maintained by the FHCRC. These cell lines and the future panels created from them, can be used by subsequent workshops or individual laboratories as an unambiguous reference when evaluating genotyping performance, or can be used to identify discrepancies obtained from the various reagent/platforms being evaluated. The corresponding IHWG Number, IHWG Sample ID and the number of laboratories that performed testing on each cell line is available in Supplemental Table 1.

Abbreviations (list of abbreviations used)

CWD

Common and Well Documented

FHCRC

Fred Hutchinson Cancer Research Center

FTP

File Transfer Protocol

GL

Genotype List

IHIW

International HLA and Immunogenetics Workshop

IHWG

International Histocompatibility Working Group

NGS

Next Generation Sequencing

PS

Pilot Study

PT

Proficiency Testing

QC

Quality Control

RP

Reference Panel