Histological validation of a type 1 diabetes clinical diagnostic model for classification of diabetes

Abstract

Aim

Misclassification of diabetes is common due to an overlap in the clinical features of type 1 and type 2 diabetes. Combined diagnostic models incorporating clinical and biomarker information have recently been developed that can aid classification, but they have not been validated using pancreatic pathology. We evaluated a clinical diagnostic model against histologically defined type 1 diabetes.

Methods

We classified cases from the Network for Pancreatic Organ donors with Diabetes (nPOD) biobank as type 1 (n = 111) or non-type 1 (n = 42) diabetes using histopathology. Type 1 diabetes was defined by lobular loss of insulin-containing islets along with multiple insulin-deficient islets. We assessed the discriminative performance of previously described type 1 diabetes diagnostic models, based on clinical features (age at diagnosis, BMI) and biomarker data [autoantibodies, type 1 diabetes genetic risk score (T1D-GRS)], and singular features for identifying type 1 diabetes by the area under the curve of the receiver operator characteristic (AUC-ROC).

Results

Diagnostic models validated well against histologically defined type 1 diabetes. The model combining clinical features, islet autoantibodies and T1D-GRS was strongly discriminative of type 1 diabetes, and performed better than clinical features alone (AUC-ROC 0.97 vs. 0.95; P = 0.03). Histological classification of type 1 diabetes was concordant with serum C-peptide [median < 17 pmol/l (limit of detection) vs. 1037 pmol/l in non-type 1 diabetes; P < 0.0001].

Conclusions

Our study provides robust histological evidence that a clinical diagnostic model, combining clinical features and biomarkers, could improve diabetes classification. Our study also provides reassurance that a C-peptide-based definition of type 1 diabetes is an appropriate surrogate outcome that can be used in large clinical studies where histological definition is impossible.

Introduction

Correct classification of diabetes type is crucial for appropriate management reduction of long-term complications. A fundamental difference between type 1 and type 2 diabetes is that the former is characterized by rapid progression to endogenous insulin deficiency due to autoimmune β-cell destruction. This difference forms the basis of differences in their treatment and management [1–3], however, this aetiopathological definition is difficult to apply in clinical practice.

Clinical features are predominately used for classification of diabetes type, with only age at diagnosis and BMI having evidence for clinical utility at onset [4]. Rising obesity rates and type 2 diabetes in young people, and the incidence of type 1 diabetes throughout life [5–7] mean that misclassification of diabetes is common, occurring in 7–15% of cases [4]. Although measurement of islet autoantibodies can assist classification, they are not perfectly discriminatory as some people with type 1 diabetes do not have islet autoantibodies and although relatively rare, autoantibodies positivity can occur in type 2 diabetes [8]. Type 1 diabetes genetic risk scores (T1D-GRS) have recently been shown to assist in discriminating between type 1, type 2 and other forms of diabetes in research settings [9,10]. Studies such as the SEARCH for Diabetes in Youth have developed classification criteria that are helpful in guiding diabetes classification at diagnosis and have informed international guidelines [11], but a difficulty with all of these studies is which standard to validate against, and that current guidelines are unable to provide simple criteria that will always ensure correct diagnosis [1–3].

We have shown previously that both clinical features [12] and biomarkers, such as autoantibodies and T1D-GRS, are most discriminative of diabetes type when combined and modelled continuously in diagnostic models that can be made widely available as an app or web calculator [4,9,13]. These models were developed and validated on C-peptide-defined type 1 and type 2 diabetes, representing differences in endogenous insulin secretion between the two types. A pilot version of our recently published model is available online (https://www.diabetesgenes.org/t1dt2d-prediction-model/). Measurement of C-peptide allows robust diagnosis of type 1 diabetes in long-standing diabetes (> 3 years’ duration) and closely relates to treatment requirements [14]. A strength of using C-peptide as an outcome is that, irrespective of any assumptions about aetiology, progression to low C-peptide associates very strongly with insulin requirement.

An alternative ‘gold standard’ would be pancreatic histology, informed by internationally accepted histological criteria [15]. Many other human diseases use histology as a gold standard, but this is not available in living people with diabetes due to the dangers of pancreatic biopsy [16]. The Network for Pancreatic Organ donors with Diabetes (nPOD) is a unique collection of human pancreata from organ donors with and without diabetes, including those with type 1 and type 2 diabetes, as well as autoantibody-positive donors without diabetes [17]. Using the nPOD biobank tissues and associated metadata, we sought to validate the performance of a previously developed clinical diagnostic model against histologically defined insulin deficiency defining type 1 diabetes. It has never been possible to validate diabetes classification against histology, and we aimed to take advantage of the nPOD biobank tissues and associated metadata to define a histological outcome which we have used to support findings from clinical studies of living patients.

Research design and methods

We assessed the performance of our previously developed diagnostic model based on clinical features (age at diagnosis and BMI) and biomarker data [islet antigen 2 (IA2) and glutamic acid decarboxylase (GAD) antibody status and T1D-GRS] in a histologically defined cohort of type 1 and non-type 1 diabetes from the nPOD biobank. We compared model performance with the performance of individual clinical features and biomarkers.

Study cohort

We identified 221 nPOD diabetes cases with native pancreas available and complete nPOD online pathology. We excluded four cases with known monogenic forms [18] and 11 with secondary causes of diabetes, because the model was designed to discriminate type 1 diabetes from type 2 diabetes. We excluded 53 cases due to incomplete biomarker or clinical information (BMI, age at diagnosis, IA2 and GAD antibody status, T1D-GRS). We categorized diabetes and analysed diagnostic model performance in the remaining 153 cases (Fig. 1). Clinical history, histopathology notes and slide digitization were available through nPOD as described previously [17]. A summary of characteristics for this cohort is shown in Table 1.

FIGURE 1.

Flow diagram of histological cohort identification from nPOD diabetes cases, excluding known monogenic forms and secondary causes of diabetes. All cases included had age at diagnosis, BMI, glutamic acid decarboxylase (GAD) antibody and islet antigen 2 (IA2) antibody status, and type 1 diabetes genetic risk score (T1D-GRS) recorded.

*Excluding known monogenic forms (18) (n=4) and secondary causes of diabetes (n=11).

Table 1.

Characteristics of histologically defined cohort

	Non-type 1 diabetes (n = 42)	Type 1 diabetes (n = 111)
BMI (kg/m2)	29.9 [27.5; 34.3]	24.3 [22; 26.6]
Age at onset (years)	37.5 [26.8; 52.3]	11.5 [6.25; 17.3]
Diabetes duration (years)	10 [3; 1]	12 [6; 23]
Age of death (years)	48.2 [40; 59.3]	27.6 [19.5; 37.1]
Sex
Female	20 (48)	51 (46)
Male	22 (52)	60 (54)
Genetic risk score	0.23 [0.21; 0.26]	0.27 [0.25; 0.29]
C-peptide (pmol/l)	1037 [429; 2072]	< 17* [< 17*; < 17*]
Antibodies†
0	38 (91)	56 (51)
1	4 (10)	32 (29)
2	0 (0.0)	10 (9)
3	0 (0.0)	13 (12)
Race
African American	12 (29)	11 (10)
Asian	2 (4.8)	0 (0.0)
White European	20 (48)	91 (82)
Hispanic/Latino	8 (19)	9 (8.1)

Values are shown as median [25th; 75^th percentiles] or n (%).

^*Limit of detection.

^†Islet autoantibodies counted include glutamic acid decarboxylase antibodies, islet antigen 2 antibodies, and ZnT8A. mIAA is not included in this count as it is not a reliable marker of autoimmunity in persons receiving exogenous insulin.

Histological definition of type 1 and non-type 1 diabetes

We categorized diabetes as type 1 (n = 111) or non-type 1 (n = 42) using visualization of digitized slides via nPOD online pathology database and/or nPOD pancreas material held in Exeter, which were stained for the presence of insulin and/or glucagon using standard immunohistochemical approaches, as described previously [19,20]. Slides were double-stained for insulin/glucagon, or serial sections were stained for insulin and glucagon respectively, where alignment of the two allowed identification of insulin-deficient islets. Histology was reviewed by two independent investigators in Exeter. A minimum of two slices per pancreas section (head, body or tail) per donor was reviewed. We defined type 1 diabetes histologically by the lobular loss of insulin-containing islets with the presence of multiple (> 10) insulin-deficient islets. Non-type 1 diabetes was defined as having no insulin-deficient islets across all viewed sections of the pancreas [15]. Islets were defined as having > 10 insulin-and/or glucagon-positive cells. As there is no internationally agreed definition of type 2 diabetes, we did not attempt to positively classify type 2 diabetes on histology.

Autoantibody measurement

Autoantibody positivity status was measured by nPOD (Organ Procurement Organizations screening laboratories) using a modified rapid enzyme-linked immunosorbent assay (ELISA) kit (Kronus, Star, ID, USA) with internal calibration on donor serum. Autoantibody-positive samples were re-analysed with an ELISA kit (Kronus, Gainesville, FL, USA), and at the nPOD autoantibody core for GAD antibody, IA2 antibody, mIAA and ZnT8A by radioligand-binding assay (Denver, CO, USA) [21] as described previously [22].

C-peptide measurement and DNA isolation

Sera were obtained during the donor-screening process and/or at donor organ recovery. Donor C-peptide was determined at the Northwest Lipid Metabolism and Diabetes Research Laboratories (S. Marcovina, University of Washington, Seattle, WA, USA) by a two-site immuno-enzymometeric assay using a Tosoh 2000 auto-analyser (TOSOH, Biosciences, Inc., San Francisco, CA, USA). C-peptide levels are reported in pmol/l with 1000 pmol/l = 3 ng/ml. We did not perform a primary analysis against C-peptide as an outcome because of the interaction between renal failure (frequent in organ donors) and sample storage time (also less controlled in organ donors). DNA was extracted from frozen spleen where available [17] and analysed for type 1 diabetes genetic susceptibility on a UFDIchip Axiom genotyping array (ThermoFisher Scientific) as described below.

T1D-GRS generation

The T1D-GRS was generated using 30 single nucleotide polymorphisms (SNPs) either genotyped directly (n = 26) or imputed (n = 4, imputation r² > 0.90) from a custom UFDIchip Axiom genotyping array from ThermoFisher Scientific. In total, the array covers 974 650 unique variants. UFDIchips were processed on an Affymetrix Gene Titan instrument with external sample handling on a BioMek FX dual arm robotic workstation. Genetic data underwent standard quality control procedures at the SNP, sample and plate levels using Axiom™ Analysis Suite 3.0 (ThermoFisher Scientific) set to default stringency thresholds as recommended. Next, discrepancies were assessed for genotyped HLA vs. imputed four-digit HLA (Axiom™ HLA Analysis software), as well as for genetic vs. reported sex. Samples that failed QC or were discordant were discarded. Finally, samples were imputed to the Human Reference Consortium (version r1.1) using Michigan Imputation Server [23]. T1D-GRS was calculated on the nPOD cohort as described previously [9,24] and indicates type 1 diabetes risk as a continuous variable.

Combined diagnostic model

We calculated the probability of type 1 diabetes on all 153 included cases using our previously developed diagnostic model [13] (Table S1). We assessed performance of the model against histologically defined type 1 diabetes in the nPOD cohort. We tested the previously developed clinical diagnostic model in four combinations:

1. Clinical features only (age at diagnosis + BMI);

2. Clinical features + T1D-GRS;

3. Clinical features + IA2 antibody + GAD antibody;

4. Clinical features + IA2 antibody + GAD antibody + T1D-GRS.

The primary analysis was to assess the discriminative power and calibration of the diagnostic model in nPOD. We carried out a secondary sensitivity analysis in a white European ancestry subgroup of the cohort diagnosed at between 18 and 50 years of age, in line with the inclusion criteria of the original model development cohort [13] (N = 31, type 1 diabetes n = 19; Table S2).

All procedures were in accordance with federal guidelines for organ donation and approved by the University of Florida Institutional Review Board.

Statistical methods

We assessed discriminative performance by estimating the area under the curve of the receiver operator characteristic (AUC-ROC). We used the integrated discrimination improvement index (IDI) [25] to assess improvements in discrimination slopes when adding in additional features. Calibration was assessed by comparing observed proportions against predicted probabilities using calibration plots and the Brier score, where a score of 0 indicates that the model is completely accurate. We tested for statistical evidence of miscalibration using the Spiegelhalter z-test (P < 0.05 representing evidence of miscalibration). All AUC-ROC analysis was performed using the pROC package in R and AUC estimated with DeLong’s algorithm. We used a two-tailed DeLong comparison of ROC curves to test for significant improvement in discriminative power against the clinical features only model. Calibration analysis and statistics were performed using the Hmisc and rms packages in R.

Results

Individual clinical features or biomarkers are discriminative of type 1 diabetes

Age at diagnosis, BMI, autoantibodies (GAD and IA2) and T1D-GRS were all strong individual discriminators of type 1 diabetes when modelled continuously (Fig. 2). The discrimination varied from an AUC-ROC of 0.71 for autoantibodies to 0.93 for age at diagnosis. This highlights that no single feature in isolation predicted histology perfectly.

FIGURE 2.

Comparative discrimination of type 1 diabetes and non-type 1 diabetes cases from the nPOD biobank. Receiver operating characteristic (ROC) curve and corresponding area under the curve (AUC) statistics and distribution are shown for BMI (A,B), age at diagnosis (C,D), autoantibody count (E,F) and type 1 diabetes genetic risk score (G,H).

Type 1 diabetes clinical diagnostic model validates well against a histological gold standard

All combinations of the type 1 diabetes clinical diagnostic model tested validated well against a histological definition of type 1 diabetes. Model combination 4, using clinical features continuously with the addition of IA2 and GAD antibody status, as well as T1D-GRS offers better discrimination than a model using clinical features only [AUC-ROC = 0.97, 95% confidence interval (CI) 0.95–1.00 vs. 0.95, 95% CI 0.91–0.98; P = 0.03] (Fig. 3). Addition of either IA2 and GAD antibody status or T1D-GRS improved the discrimination slope (IDI = 0.05, 95% CI 0.01–0.08; IDI = 0.07, 95% CI 0.02–0.12) (Fig. S1).

FIGURE 3.

The discriminative ability of diagnostic model 4 combining BMI, age at diagnosis, autoantibody status and type 1 diabetes genetic risk score (T1D-GRS) to identify type 1 diabetes cases. Receiver operating characteristic (ROC) curve and corresponding area under the curve (AUC) statistics (A). A boxplot of model 4 predicted probabilities of type 1 diabetes (B).

The type 1 diabetes clinical diagnostic model calibrates well

The mean overall probabilities of type 1 diabetes in the nPOD cohort for each combination of clinical diagnostic model tested closely reflected the proportion of observed type 1 diabetes cases in the study (111 of 153, 73%) (Fig. S2) indicating overall good calibration. We found no evidence of miscalibration across all model combinations as indicated by a low Brier score (B = 0.06–0.08) and non-significant Spiegelhalter z-statistics (Z < 1.76) (Table S3).

Sensitivity analysis in white European subgroup diagnosed in adulthood (18–50 years of age)

Results of a sensitivity analysis, using a white European ancestry subgroup diagnosed at between 18 and 50 years of age, showed equivalent discriminatory power for all variations of the type 1 diabetes clinical diagnostic model (N = 31, type 1 diabetes = 19, AUC-ROC > 0.84) (Fig. S3). A summary of characteristics for this subgroup is shown in Table S2.

Characteristics of cases with discordant model classification compared with histology

The distribution of probabilities of type 1 diabetes generated by model combination 4 are outlined in Fig. 3(B). This highlights that a clinical diagnostic model will give an output that is a continuous distribution of probabilities, with a small number of type 1 diabetes cases still having low probability of type 1 diabetes and some without type 1 diabetes still identified as having a high probability. We examined the features of cases that had probabilities at the extreme distributions of model combination 4: two cases with histological type 1 diabetes who had a probability of type 1 diabetes < 25%; and three cases with histological non-type 1 diabetes who had a probability of type 1 diabetes > 75%. The characteristics of these cases are outlined in Table S4. Serum C-peptide levels in these cases matched the histological classification (two with histological type 1 diabetes had C-peptide < 30 pmol/l, and three with histological non-type 1 diabetes had C-peptide > 1000 pmol/l). Despite our concerns about C-peptide storage and sampling in organ donors, the observed serum C-peptide levels in type 1 vs. non-type 1 diabetes in the whole cohort was significantly different [median < 17 pmol/l (limit of detection) vs. median 1037 pmol/l; P < 0.0001) (Table 1).

Discussion

This is the first study to evaluate a clinical diagnostic model against histological data. We have demonstrated that a model developed previously to classify type 1 diabetes defined by insulin deficiency, is discriminative of type 1 diabetes when using a histological outcome, not possible in routine clinical care. We found that using a combined model performed better than individual clinical features and biomarkers in discriminating type 1 diabetes and non-type 1 diabetes donor cohorts. Our study contributes to the evidence that diagnostic models combining clinical features with at least one clinical biomarker could assist classification of diabetes in clinical practice, and is already available as a beta-version online (https://www.diabetesgenes.org/t1dt2d-prediction-model/).

We previously demonstrated that a classification model, which integrated genetic testing combined with multiple continuous clinical variables, was effective at discriminating maturity-onset diabetes of the young (MODY) from type 1 diabetes [12]. An advantage in identification of MODY is that the outcome, a genetic mutation causing diabetes, is often definitive, but there is less clarity on a standard definition of type 1 and type 2 diabetes. In developing diagnostic models for diabetes classification, we used progression to insulin deficiency, as measured by serum C-peptide in long-standing diabetes (> 3 years’ duration), as a surrogate marker of type 1 diabetes [9,13]. We assumed that insulin deficiency, as defined by serum C-peptide < 200 pmol/l at > 3 years post diagnosis, was an accurate surrogate of type 1 diabetes [14]. This study provides evidence that this assumption is valid, by showing that our model developed on clinical data to predict C-peptide deficiency near perfectly reflects histologically defined insulin deficiency (a robust but rarely used definition of type 1 diabetes). This result is further reinforced by comparison of C-peptide in type 1 and non-type 1 diabetes groups, which was non-overlapping (Table 1). Clinically, one strength of a model trained on severe insulin deficiency as an outcome is that prediction of severe insulin deficiency has a clear treatment implication, the requirement of exogenous implications.

We used histological criteria for type 1 diabetes based on work by Campbell-Thompson et al. [15]. Our criteria focus on insulin deficiency and the presence of insulin-deficient islets as a hallmark of type 1 diabetes that is present in all type 1 diabetes cases. The international consensus definition of type 1 diabetes histology describes various exclusive pathological features in the pancreas. These include the presence of insulitis that is always accompanied by pseudoatrophic islets devoid of β cells [15]. However, the proportion of inflamed islets declines over time such that it is seen most readily in short duration type 1 diabetes donors (< 1 year) [26]. As the majority of the nPOD donors had a longer duration of disease, and the presence of insulin-deficient islets is evidence of prior insulitis; we used the detection of insulin-deficient islets as our key histopathological criterion to define type 1 diabetes in this study.

We focused on the positive histological definition of type 1 diabetes rather than defining other diabetes types by histology, and excluded cases that had a diabetes diagnosis of monogenic diabetes or secondary causes of diabetes. The clinical features of our non-type 1 diabetes group suggest that this group is composed of predominantly type 2 diabetes, however, there is much less consensus on the histology of other diabetes types, including type 2 diabetes, and our original model was designed with features that discriminate type 1 from type 2 diabetes, such as age at diagnosis and BMI. In the future it may be possible to develop an approach that additionally classifies type 2 diabetes and less-common diabetes types. This will require larger collections of non-type 1 diabetes cases [27] to allow accurate characterization of type 2 diabetes pancreatic features.

A notable limitation of our study is that the current diagnostic model was developed using data derived primarily from white Europeans between the ages of 18 and 50 years. It is well documented that the incidence and prevalence of type 1 and type 2 diabetes vary across demographic subgroups [28,29]. It is also well accepted that the prior prevalence of type 1 and type 2 diabetes varies with age, with type 2 diabetes more likely to be diagnosed at older ages and type 1 diabetes more likely to be diagnosed at younger ages. Our cohort included 27% non-white Europeans and age at diagnosis ranged from 1 to 73 years, yet despite this, the model showed good discrimination and calibrated well overall (Table S3). Owing to the limitations of the sample size in our study, further validation evidence of the model performance is still required in non-white Europeans, in children, and in adults over the age of 50. It is likely that the model will need to be further refined for these age groups.

Our analysis used some features that are unchanged at diagnosis (age at diagnosis and T1D-GRS), but other features that were recorded at the time of organ donation and could theoretically have been different at the time of diagnosis (autoantibody status, BMI). Despite this, both BMI and autoantibodies were discriminative. We hypothesize that the discriminative power of these two variables will only be enhanced by ascertainment at the time of diagnosis, further improving model performance. It is possible that, at diagnosis, a model with only three variables (e.g. age at diagnosis, BMI and one of either autoantibodies or T1D-GRS) will perform as well as a four-variable model. It will be impossible to test this in studies of organ donors, but we are currently testing this in a prospective study assessing clinical features and biomarkers at the time of diagnosis (ClinicalTrials.gov identifier: NCT03737799). Our sample size limited our ability to test if a model using all four variables was significantly superior to a model using either T1D-GRS or autoantibodies (Fig. S1C–F). Existing work suggest a three-variable model with either autoantibodies or T1D-GRS is as good as a model with four variables [9,13]. It is likely that the relative benefits of autoantibody testing (a routinely available clinical test that is very discriminative if taken at diagnosis) [8] and T1D-GRS (time-independent and freely available in population biobanks) [30] will see them used differently depending on the setting and availability. We did not have some potentially relevant features at diagnosis, such as the presence of ketoacidosis and pre-diagnosis weight loss, but to date these have not been shown to be reliable discriminators of type 1 diabetes [4]. However, it will require larger studies with detailed information at diagnosis, across diverse ages and ethnicities, to fully elucidate the most accurate method and combination of features to classify diabetes at diagnosis.

Despite the modest sample size of our study, limited by the numbers of organ donors available worldwide, our study provides robust histological evidence that a model combining clinical features and biomarkers offers improved discrimination of type 1 diabetes, and that progression to C-peptide deficiency is an appropriate surrogate endpoint. Our study therefore provides further evidence for a clinical diagnostic model having utility to identify type 1 diabetes in clinical practice, and for C-peptide as a surrogate outcome for clinical studies in which histological classification is not possible. Overall the study strengthens the evidence that a clinical diagnostic model may aid classification in clinical practice.

Supplementary Material

supp info

Figure S1. Comparative discriminative ability of diagnostic model variations 1, 2 and 3 to identify type 1 diabetes cases.

Figure S2. Calibration plots for four variations of a clinical diagnostic model 1–4.

Figure S3. The comparative discriminative ability for all variations of the diagnostic model to identify type 1 diabetes cases in a white European subset diagnosed 18–50 years of age.

Table S1. Regression equations from a combined prediction model developed in a separate clinical cohort.

Table S2. Characteristics of white European subset diagnosed 18–50 years of age of histologically defined type 1 diabetes cohort.

Table S3. Comparison of calibration statistics for clinical diagnostic model, model combinations 1–4.

Table S4. Characteristics of cases with discordant model 4 prediction and histological classification.

What’s new?

• Misclassification of diabetes at diagnosis is common due to an overlap in the clinical features of type 1 and type 2 diabetes.

• Combining clinical features and biomarkers in a diagnostic model improved discrimination of diabetes type, defined by insulin deficiency (measured by C-peptide assays), over use of any single characteristic.

• No diabetes classification studies have used pancreatic histology to define type 1 diabetes.

• A diagnostic model, developed using diabetes type defined by C-peptide level as an outcome, validates against histologically defined insulin deficiency.

• C-peptide provides a robust surrogate definition of type 1 diabetes that can be used in diagnostic model development.

• Our study provides the first histological evidence for a clinical diagnostic model having utility to identify type 1 diabetes in clinical practice.