Multiparametric Quantitative Imaging Biomarkers for Phenotype Classification: A Framework for Development and Validation

Abstract

This manuscript is the third in a five-part series related to statistical assessment methodology for technical performance of multi-parametric quantitative imaging biomarkers (mp-QIBs). We outline approaches and statistical methodologies for developing and evaluating a phenotype classification model from a set of multiparametric QIBs. We then describe validation studies of the classifier for precision, diagnostic accuracy, and interchangeability with a comparator classifier. We follow with an end-to-end real-world example of development and validation of a classifier for atherosclerotic plaque phenotypes. We consider diagnostic accuracy and interchangeability to be clinically meaningful claims for a phenotype classification model informed by mp-QIB inputs, aiming to provide tools to demonstrate agreement between imaging-derived characteristics and clinically established phenotypes. Understanding that we are working in an evolving field, we close our manuscript with an acknowledgement of existing challenges and a discussion of where additional work is needed. In particular, we discuss the challenges involved with technical performance and analytical validation of mp-QIBs. We intend for this manuscript to further advance the robust and promising science of multiparametric biomarker development.

1. Introduction

Quantitative imaging biomarkers (QIBs) are objectively measured, ratio or interval scale characteristics derived from one or more in vivo images that indicate a normal biological process, a pathogenic process, or a response to a therapeutic intervention¹. In a 2015 metrology series^1–6 the Quantitative Imaging Biomarker Alliance (QIBA) reviewed statistical methods for characterizing the analytical / technical performance of QIBs. Multiple QIBs, when combined to form a multi-parametric QIB (mp-QIB), promise additional clinical utility over single quantitative imaging biomarkers for characterizing tissue, detecting disease, identifying phenotypes and longitudinal change, predicting outcomes, or other potential intended uses. Multiple QIBs can be selected from within single or multiple imaging modalities to represent different aspects of tissue morphology or biology. If selected carefully, each QIBs has the potential to contribute unique clinical information, and the value of the mp-QIB exceed that of its individual components.

In a new series of papers, QIBA identifies four use cases for mp-QIBs⁷. For each use case, a framework for the development and validation of the relevant mp-QIBs is described. In the first use case, a vector of p QIBs X = (X₁, …, X_p) is treated as a p-dimensional multivariate descriptor of health⁸. This manuscript discusses the second use case: combining X₁, …, X_p to differentiate between phenotypes. The third use case involves combining X₁, …, X_p to forecast the risk of occurrence of an event of interest, such as death, disease progression, or recurrence⁹. Finally, the fourth use case describes the applicability of these frameworks to radiomic data¹⁰.

Phenotypes are observable characteristics of a subject, including those arising from response to the environment¹¹. Statistically, a phenotype can be one of three types of categorical variables: (1) dichotomous, (2) nominal polychotomous--that is, more than two possible values that cannot be meaningfully ordered--and (3) ordinal polychotomous--that is, more than two possible values that can be meaningfully ordered but are not quantitative, i.e., not interval or ratio scale. In metrology and medicine, ordinal variables are called “semi-quantitative”. Examples of a dichotomous phenotype variable are presence or absence of a disease and benign or malignant classification of a lesion or nodule¹². Examples of a nominal, polychotomous phenotype variable are stroke subtype¹³, skin lesion type¹⁴, and breast cancer subtype¹⁵. Examples of a semi-quantitative (ordinal) phenotype variable are liver fibrosis stage¹⁶, histopathological classification of cervical intraepithelial neoplasia¹⁷, and prostate Gleason grade¹⁸ (Table 1). While phenotypes can be identified from an individual feature, for example vascularity (through dynamic contrast enhanced studies), glycolytic metabolism (through standardized uptake values of glucose on 18F-FDG PET scans) or elasticity (through measurements of shear wave on ultrasound or MRI), the subject of this manuscript is combining multiple quantitative measurements into a defined phenotype.

Table 1 –

Examples of Phenotype Classification Tasks

Phenotype Classification Task	Definition	Examples
Dichotomous	Two categories	Malignant/benign classification Presence/absence of disease
Nominal polychotomous	Three or more categories, not ordered	Atherosclerotic Plaque Type Classification (subclinical/stable/unstable) Blood type (A, B, O, AB) Breast lesion classification (benign, mass, calcification, architectural distortion, etc.) Stroke subtype (large-artery atherosclerosis, cardioembolism, small-vessel occlusion, stroke of other determined etiology, stroke of undetermined etiology) Lung Texture (normal, hyperlucency, fibrosis, ground glass, reticular, honeycomb, solid, focal, etc.)
Ordinal polychotomous	Three or more categories, ordered	Disease Severity or Staging, e.g. METAVIR liver fibrosis stage F0-F4, cancer staging (I-IV), prostate Gleason grade, Breast imaging reporting and database system (BI-RADS) score 1–6. Head CT scan for intracranial hemorrhage, mass effect, midline shift, or cranial fracture Cervical intraepithelial neoplasia (CIN) histology (normal, CIN 1, CIN 2, CIN 3, cancer) Pap smear cytology (NILM, ASC-US, LSIL, HSIL, cancer) Diagnostic tests with equivocal zones, e.g., hepatitis C test negative, equivocal, or positive.

Development of a phenotype classification model from imaging findings requires first demonstrating connection between an existing, stable phenotype and imaging findings (e.g. demonstrating biological plausibility). This discovery phase is followed by development and refinement of the model/classifier and assessment of the analytical/technical performance of the model/classifier. Evaluation of the clinical implications, sometimes termed clinical utility, of classification (e.g. prognosis prediction, treatment selection, etc.) is the fourth and final step. We focus this manuscript on model/classifier development and analytical/technical performance assessment.

We argue that diagnostic accuracy (defined here as agreement of the imaging-based classification to an established phenotype) as well as interchangeability of imaging-based classification with a comparative measure of the established phenotype are relevant and supportable claims for a multi-parametric model developed to classify phenotypes. The task we tackle in the rest of this manuscript is development of statistical methods to characterize the performance of the classifier and demonstrate agreement and/or interchangeability between imaging-derived characteristics and clinically established phenotypes. Examples of such clinical phenotype classification tasks include staging liver fibrosis or steatosis from non-invasive imaging results (rather than from pathology) or determining stability of atherosclerotic plaque from imaging findings (rather than from tissue samples). Data-driven studies intended to identify groupings within a heterogenous patient populations^19–21 may be extremely useful in identifying phenotypes and use case #4 provides guidance on such exploratory studies. Such clustering could be considered for this use case once clinical recognition of the phenotype is established. Furthermore, we focus the work of this manuscript on the multi-class situation, as a wealth of previous literature exists for the dichotomous (binary) classification case^12,22–30.

2-. Model Development

The task we tackle here is combining p QIB measurements X = (X₁, X₂, … X_p) into an mp-QIB classifier of K possible phenotypes 1, 2, …, K. For K = 2, Y is bivariate, and phenotype status is dichotomous. For K > 2, Y is multivariate, and phenotype status is polychotomous. In this manuscript, we focus on development of models for classification of K > 2 phenotypes, which can be either nominal or ordinal.

Y is approximated with a model f(X, β) = (f₁(X, β), f₂(X, β), … f_K(X, β)) that is a function of X and its associated parameter β, where 0 ≤ f_k(X, β) ≤ 1 and ∑f_k(X, β) = 1, and f_k(X, β) is the probability that the subject has phenotype k. For example, in a neural network, β includes the sets of weights and biases embedded in the input and hidden layers (see the Supplement). For classification, the maximum probability may be used to assign a subject with X = x as phenotype k = argmax_i f_i(x, β). If the cost of misclassifying phenotype j as k depends on j and k, then the subject may be assigned to the phenotype that minimizes expected cost (Supplement).

The cost or loss function L(Y, f(X, β)) measures the error of approximating Y with f(X, β). For classification, L(Y, f(X, β)) is often taken to be the squared error or cross-entropy (deviance) error function³¹. The expected value of L(Y, f(X, β)) is the risk

$$\begin{array}{l}R(\beta )=\int L\left(Y,f\left(X,\beta \right)\right)dP\left(X,Y\right)\\ =\int \left[\int L\left(Y,f\left(X,\beta \right)\right)dP\left(X,Y\right)\right]dP\left(X\right),\end{array}$$

where P(X, Y) = P(X)P(Y|X) is the joint distribution of X and Y, P(Y| X) is the conditional distribution of Y given X, and P(X) is the marginal distribution of X.

When f is specified, the goal is to minimize R(β) over the class of functions f(X, β), β ∈ B, which can be achieved by minimizing ∫ L(Y, f(X, β))dP(Y| X) for every X ³². In many machine learning approaches, the form of f is left unspecified and R(β) is minimized over both f and β. The model is developed by fitting f(X, β) to training data. Typically the data consist of random independent identically distributed observations {(x₁, y₁), … (x_i, y_i), …, (x_n, y_n)} on n subjects, where, for subject i, $x_i=\left(x_{i1},\mathrm{...},x_{ip}\right)\prime$ is the set p QIB values and y_i is the phenotype. To guard against overfitting, the algorithm for minimizing R(β) can be stopped early or a regularization technique imposed, among possible options (Supplement). When f is specified, the fitted model is denoted $\widehat{Y}=f\left(X,\widehat{\beta }\right)$, where $\widehat{\beta }$ is the estimate of β that minimizes the sample data version of R(β). If f as well as β is estimated, then the fitted model is $\widehat{Y}=\widehat{f}\left(X,\widehat{\beta }\right)$.

In general, the model output $\widehat{Y}=\widehat{f}\left(X,\widehat{\beta }\right)$ need not be probabilities of classification, but instead can be a continuous numeric score to which K − 1 thresholds are applied to obtain a classification of each subject into one of the K phenotypes. For example, subjects have been classified into one of the four METAVIR liver fibrosis stages F0–1, F2, F3, and F4 based on applying three thresholds to continuous ultrasound measurement of shear wave elastography (SWE) or transient elastography (TE)³³.

For classification, $\widehat{Y}$ is typically either an assignment of X to one of the phenotypes or a set of probabilities of each phenotype. Specific to this use case, we note that while mp-QIB model inputs may be QIBs, the output of phenotype classification is not a QIB.

2.1. Model Inputs

Parameters must have biological plausibility to be candidate model inputs. However, this connection alone is insufficient, and the behavior of model inputs should also be well characterized, as stability of model inputs is necessary to inform development of a robust and reliable model. At minimum, repeatability and reproducibility of candidate model inputs should be assessed. Methods for assessing repeatability and reproducibility of a QIB are described in previous literature^34,35. The more desirable state is to include fully validated QIBs as model inputs.

Highly correlated QIBs should be avoided as model inputs, as high correlations are potentially an indication that the QIBs measure the same thing and only one is needed. As a general rule, when correlation between two QIBs exceed 0.8, only one QIB should be selected for inclusion in the model.

Additional items to consider when developing models are the incremental value of each model input and whether the addition of a new parameter adds value to the model. Change in the area under the receiver operating characteristic curve (AUC is the same as the c-index when outcomes are binary), integrated discrimination improvement (IDI) and continuous net reclassification improvement (NRI) are all available metrics^36–39. Change in AUC with and without the inclusion of the additional parameter is perhaps the most commonly used. An improvement in AUC for a model containing the new marker provides evidence that the marker adds value; the criticism of this approach being that the incremental value of the additional marker must be quite large to result in a meaningfully larger AUC. Beyond change in AUC, Pencina et.al. propose Net Reclassification Index (NRI) and Integrated Discrimination improvement (IDI) as useful metrics to measure improvements in reclassification⁴⁰. Pepe et.al. emphasize the importance of defining a minimal magnitude of improvement and testing of that rather than simply the presence of any improvement³⁸.

2.2. Different Types of Models

The three main categories of classification approaches--classical statistical methods, machine learning approaches and deep learning neural networks—are discussed in the supplemental material.

3. Model Validation

Validation of the outputs of a developed classification model is a necessary and vital step before deployment of the model.

Internal validation is often aimed to optimize model performance and is conducted using a dataset that is not fully independent from the development dataset. Common procedures for internal validation of a model include leave-k-out cross-validation^41,42 and bootstrap cross-validation^43,44. A QIB developed on a training dataset will be fit to measurement error patterns and other peculiarities in the dataset as well as actual signal. Internal validation of the QIB on the same dataset on which it was trained, even if done properly, will reflect those peculiarities and thus may not be robust enough for the observed performance to reproduce in new subjects. Thus internal validation is generally not enough to establish a data-driven model⁴⁵.

External validation in independent datasets is a necessary component of rigorous model assessment. The aim of external validation is to generalize and establish performance of the model in a study population different than was used for model development. External validation is performed on the locked-down model in a dataset independent of the dataset(s) used to develop the model (i.e. collected at clinical sites different from the training dataset)⁴⁵.

While the external evaluations of multiparametric QIB phenotype classifiers will ultimately depend on the intended use of the model and the data available for model development, three basic performance evaluations are considered further in this manuscript:

• Precision. Closeness of agreement of replicate classifications obtained on the same object under specified conditions⁴⁶.

• Diagnostic Accuracy. Agreement of a classification method with the true (established) phenotype, as determined by a clinical reference standard⁴⁷ (which for imaging classifiers is a method other than imaging).

• Interchangeability. Two classification methods agree such that one can be swapped for the other with negligible effect on diagnostic accuracy and clinical benefit.

Diagnostic accuracy and interchangeability are clinically meaningful claims for a multi-parametric QIB model developed to classify phenotypes. An imaging-based classification is accurate if it agrees with the established (true) phenotype. An imaging-based classification is interchangeable with a comparative classification if the two classifiers can be exchanged in clinical practice without a change in the clinical management of the patient. While bias and imprecision are important to characterize, as a clinical accuracy study of an imprecise classifier is likely to fail, the clinical interpretation of bias or imprecision for a phenotype classifier is not entirely transparent. Thus, while important to characterize, a claim that a phenotype classifier meets a specific bias or imprecision goal is not recommended to be sufficient to support clinical use of the classifier.

4. Statistical Methods for Evaluating Classifier Outputs

A phenotype classification model can be designed to classify subjects into one of the phenotypes or to output the probability of each phenotype. Methods for evaluating probabilities of having two possible phenotypes either at present or in the future are discussed in Use Case 3⁹. In this use case, we provide a non-comprehensive review of methods for evaluating precision, diagnostic accuracy, and interchangeability of phenotype classifiers, focusing our effort on the scenario of K>2 phenotypes.

4.1. Precision

Repeatability is the precision (closeness of agreement) of repeated measures when conditions of testing are held constant⁴⁶. Reproducibility is the precision of repeated measures when specific conditions of testing are varied⁴⁶. For quantitative measurements, e.g., QIBs, imprecision is commonly characterized by the within-subject standard deviation (wSD) or coefficient of variation (CV) of repeated measurements⁴⁶. The reproducibility coefficient (RDC) is defined as $1.96\sqrt{2}$ times wSD^3–5,48. If a serial change in a quantitative measurement is greater than its RDC, then it is likely not explainable by measurement imprecision but in fact a real change in the measurand (under stated conditions). However, for phenotype classification, SD and CV do not make sense because phenotype labels (e.g., fibrosis stage) do not express distance, that is, do not constitute an interval or ratio scale variable.

4.1.1. Variation in Categorical Data

For a phenotype classification variable S with categories i = 1,2, 3, …, K, one measure of the variation is the Gini concentration index⁴⁹, $g\left(\mathit{\tau }\right)=1-{\sum }_{i=1}^K{\tau }_i^2$, where τ = (τ₁, τ₂, . . . τ_K) and τ_i = Pr(S = i). For K = 2 with τ = Pr(S = 2),

$$g\left(\tau \right)=1-{\tau }^2-{\left(1-\tau \right)}^2=2\tau \left(1-\tau \right),$$

or twice the Bernoulli variance.

Clinically, g(τ) is interpretable as the probability that two realizations of S fall into different categories. In precision experiments of repeated classifications taken on the same subject, g(τ) is the probability that two repeated classifier results fall into different categories.

The minimum of g(τ) is 0 when τ_k = 1 for one category k and τ_i = 0 for all other categories i ≠ k. The maximum value of g(τ) is (K − 1)/K when τ_i = 1/K for all i = 1,2, … K.

The sample estimate of the Gini index, its variance, and an example of their calculation are given in the supplementary material. Two other measures of categorical variation are misclassification error⁵⁰ and entropy⁵¹ (Supplement).

4.1.2. Test-Retest Precision Study

To evaluate precision on subjects, a test-retest study could be conducted in which two or more repeated measures are obtained per subject over a short period of time. Variation between the repeated measures within each subject could be characterized, then pooled across subjects to obtain an overall precision estimate. For example, for QIBs, the within-subject standard deviation (wSD) can be estimated, then pooled across subjects to obtain an overall measure of imprecision, provided wSD can be assumed to be homogeneous across subjects with different measurand values. (For many quantitative measurements, a better assumption is that the within subject coefficient of variation is homogeneous across subjects).

For phenotype classification, pooling a measure of variability across subjects is less clear but may be possible for subjects having the same phenotype. If testing conditions will be varied according to factors (e.g., imaging manufacturer, instrument within manufacturer, imaging acquisition parameter, etc.), a balanced incomplete block (BIB) design in the factor levels could be considered in which subject is the block. With a BIB design, the contribution of each factor to the total variability can be quantified. Analysis of a test-retest study for a phenotype classifier is an exciting topic for future research.

4.2. Diagnostic Accuracy

A diagnostic test result is accurate if it agrees with the accepted reference value for the measurand. For our purposes, the measurand is the phenotype and the clinical reference standard is the best available non-imaging method of assessing the established phenotype. For example, for METAVIR fibrosis stage, histological evaluation of liver biopsy is the clinical reference standard because it is considered the best available reference method (even though liver biopsy has its own limitations, including sampling error and pathologist intra- and interobserver variability in histological interpretations).

4.2.1. Binary Test Accuracy

Binary diagnostic tests are classifiers that without loss of generality render either a positive or negative test result indicating presence or absence of a condition, in this case a phenotype. The pre-test probability of the phenotype is its prevalence. The post-test probability of the phenotype is the predictive value of the test. The positive predictive value (PPV) is the probability that the subject has the phenotype given the test is positive. The negative predictive value (NPV) is the probability that the subject does not have the phenotype given the test is negative. Conversely, sensitivity (Se) is the probability of a test positive in subjects with the phenotype, and specificity (Sp) is the probability of a test negative in subjects without the phenotype. The positive and negative diagnostic likelihood ratios PLR = Se / (1 − Sp) and NLR = (1 − Se)/Sp are, by Bayes Theorem, the relative changes from pre-test to post-test in the odds of having the phenotype conferred by positive and test negative results, respectively^27,28.

Binary classification models for presence of a phenotype are often based on an underlying continuous value X to which a threshold c is applied to obtain without loss of generality a positive test result when X > c and a negative test result when X ≤ c. For example, X could be the probability of having the phenotype to which a probability threshold of ‘c=0.05 could be applied to obtain the binary decision of whether the phenotype is absent or present. The receiver operating characteristic (ROC) curve^52,53 is the plot of Se(c) vs. 1 – Sp(c) for all thresholds c. A summary measure is the area under the ROC curve (AUC).

A minimal but insufficient requirement for a binary test to be clinically meaningful is that it is informative for, i.e., associated with, phenotype status. A test is informative if Se > 1 – Sp, PLR > 1, NLR < 1, PLR > NLR, PPV > 1 − NPV, or, for continuous numeric scores, AUC > 0.5. Performance metrics and methodologies for evaluating binary test accuracy are reviewed in many textbooks^22,23,29,30 and papers^{12,24–28,52,53}.

4.2.2. Evaluating Multi-class Classifiers with Binary Test Accuracy Measures

Binary test accuracy measures can be used to evaluate classifiers of K > 2 phenotypes. To illustrate its use, consider the study by Ferraioli et al⁵⁴ comparing real-time shear wave elastography (SWE, denoted here as S) measurements with transient elastography (TE, denoted here as T) measurements in classification of METAVIR liver fibrosis stage in 121 patients with chronic hepatitis C. The clinical reference standard for METAVIR liver fibrosis stage is histological analysis of liver biopsy (denoted here as R). SWE and TE measurements were acquired in kilopascal (kPa) units using two different ultrasound systems and thresholds were applied to the kPa measurements to create the liver fibrosis classifications.

Table 2 displays the confusion matrices of TE and SWE fibrosis stage classifications by true METAVIR fibrosis stage. For SWE, the sensitivities for fibrosis stages F0–1, F2, F3, and F4 were 87.5% (42/48), 72.7% (24/33), 84.6% (11/13), and 87.5% (21/24). For TE, the sensitivities were 89.6% (43/48), 18.8% (6/32), 53.8% (7/13), and 91.7% (22/24). Each sensitivity is accompanied by multiple specificities, one for each of the other phenotypes. For example, the sensitivity for fibrosis stage F4 is accompanied by the specificities of not being F4 when the fibrosis stages are F0–1, F2, and F3. For F4, the SWE specificities were 100% (48/48), 97.0% (32/33), and 84.6% (11/13) for F0–1, F2, and F3. If the fibrosis stage prevalences in the study are unbiased, as would be expected in a prospective study with simple random sampling of subjects, then the specificities may be pooled into an overall specificity. For SWE, the overall specificity for F4 was 96.8% (91/94 = (48 + 32 + 11)/(48 + 33 + 13)).

Table 2.

Confusion matrices of ultrasound-based shear wave elastography (SWE) and transient elastography (TE) classifications of METAVIR fibrosis stage, which was determined by histological analysis of liver biopsy (reprinted from Ferraioli et al, 2012, Table 4).

METAVIR .
SWE (kPa)	Class	F0–1	F2	F3	F4	Ttl
SWE ≤ 7.1	F0–1	42	7	0	0	49
7.1 < SWE ≤ 8.7	F2	4	24	0	1	29
8.7 < SWE ≤ 10.4	F3	2	1	11	2	16
SWE > 10.4	F4	0	1	2	21	24
Total		48	33	13	24	118
METAVIR .
TE (kPa)	Class	F0–1	F2	F3	F4	Ttl
TE ≤ 6.9	F0–1	43	19	1	1	64
6.9 < TE ≤ 8.0	F2	3	6	2	0	11
8.0 < TE ≤ 11.6	F3	2	7	7	1	17
TE > 11.6	F4	0	0	3	22	25
Total		48	32	13	24	117

Class = classification of METAVIR fibrosis stage made by the classifier

The predictive values of a classifier are the probabilities that each true phenotype agrees with the classifier result, analogous to the positive and negative predictive values of binary tests. The predictive values of SWE for F0–1, F2, F3, and F4 were 85.7% (42/49), 82.8% (24/29), 68.8% (11/16), and 87.5% (21/24). For TE, they were 67.2% (43/64), 54.5% (6/11), 41.2% (7/17), and 88.0% (22/25). Predictive values depend on the prevalences of the phenotypes, thus are unbiased only if the distribution of the true phenotypes is unbiased.

If a classifier is intended to only detect a subset of the possible phenotypes, an ‘other/none of the above’ category should be included to avoid an incomplete confusion matrix or forced misclassification. For example, if an artificial intelligence (AI) algorithm is designed to detect on chest radiograph the lung pathologies pulmonary edema (cardiomegaly), adult respiratory distress syndrome (atelectasis or pneumothorax), pneumonia (pleural effusion, consolidation, nodule), aspiration pneumonitis, or pulmonary embolism⁵⁵, subjects may obviously have other lung pathologies. Without an ‘other’ category, the algorithm would call another pathology as one of the AI-detectable pathologies and the confusion matrix reporting results would be erroneous.

4.2.3. Association Between Nominal Categorical Variables with Example

A highly accurate phenotype classifier will be highly associated with the true phenotype. A general overall measure of association is the proportion of explained variation (PEV) of one variable by another^56,57. For quantitative variables, PEV is the coefficient of determination or squared correlation between the variables. PEV can be used to quantify the association of two categorical variables. For the liver fibrosis example, let S = i be the SWE classifier result and R = j be the reference (true) phenotype where i, j = 1,2,3,4 index the fibrosis stages F0–1, F2, F3, and F4. Let τ_ij = Pr(S = i, R = j) be the joint probability and n_ij the observed count for cell (i, j) of the confusion matrix (Table 2). The total Gini variation in reference fibrosis stage R is

$$\widehat{V}\left(R\right)=g\left(\widehat{\mathit{\tau }}\right)=1-{\sum }_{j=1}^4{\widehat{\tau }}_{•j}^2=1-{\left(\frac{48}{118}\right)}^2-{\left(\frac{33}{118}\right)}^2-{\left(\frac{13}{118}\right)}^2-{\left(\frac{24}{118}\right)}^2=0.707$$

where τ_•j = Pr(R = j) and ${\widehat{\tau }}_{•j}=n_{ij}/n_{•j}$ is the sample estimate. The average Gini variation in R conditional on SWE result S is

$$\widehat{E}\left[\widehat{V}\left(R|S\right)\right]={\sum }_{i=1}^I{\widehat{\tau }}_{i•}\left[1-{\sum }_{j=1}^J{\left(\frac{{\widehat{\tau }}_{ij}}{{\widehat{\tau }}_{i•}}\right)}^2\right]=1-{\sum }_{i=1}^I\frac{{\sum }_{j=1}^J{\widehat{\tau }}_{ij}^2}{{\widehat{\tau }}_{i•}}=0.287$$

where τ_i• = Pr(S = i) and ${\widehat{\tau }}_{i•}=n_{ij}/n_{i•}$ is its sample estimate.

The proportion of Gini variation in R explained by S is therefore

$$PEV\left(g\left(\widehat{\mathit{\tau }}\right)\right)=\frac{\left\{\widehat{V}\left(R\right)-\widehat{E}\left[\widehat{V}\left(R|S\right)\right]\right\}}{\widehat{V}\left(TR\right)}=\frac{0.703-0.287}{0.703}=0.592$$

In the Supplemental Material, the proportion of variation in R explained by S is also calculated for entropy and classification error for this example. While a highly accurate phenotype classifier will be highly associated with the true phenotype, we caution that the reverse need not be true, although that would be strange to observe.

4.2.4. Association Between Ordinal Variables

A symmetrical measure of association between ordinal classifications S and R with I and J categories is Gamma^56,58. Like correlation, γ ranges from −1 to 1. A pair of subjects is considered concordant if the subject ranked higher on S also ranks higher on R. A pair of subjects is considered discordant if the subject ranked higher on S ranks lower on R. The calculation of Gamma for the liver fibrosis example is given in the supplemental material. For the SWE classification data $\widehat{\gamma }=0.9429$; for the TE classification data, $\widehat{\gamma }=0.9041$. Alternatively, for I > 2 and J > 2 ordinal categories, Obuchowski (2005)⁵⁹ proposed an AUC-like measure (Matlab code provided in Supplemental material).

4.2.5. Bias

Bias in a classifier can contribute to its inaccuracy. To evaluate bias, express the classifier result S = i as the multinomial variable S = (S₁, S₂, … S_K) with S_i = 1 and S_k = 0 for k ≠ i, and likewise express the reference (true) phenotype as R = (R₁, R₂, … R_K). The marginal distributions of S and R are τ_S = (τ_1•, τ_2•, … τ_K•) and τ_R = (τ_•1, τ_•2, … τ_•K). The multivariate bias of S for R is the difference D = τ_S − τ_R. The null hypothesis of no bias is H₀: D = 0, which is the hypothesis of marginal homogeneity of row and column proportions. For dichotomous data, the McNemar test statistic of H₀ is

$$Z=\frac{n_{12}-n_{21}}{{\left(n_{12}+n_{21}\right)}^{1/2}}$$

where n_ij is the observed count in cell (i, j) the confusion matrix. For K by K tables, marginal homogeneity can be evaluated with Stuart’s score test^56,60 which when H₀ is true asymptotically follows a standard normal distribution (mean 0, variance 1). For illustration of these tests with the SWE data, see the Supplemental Material. If the hypothesis test rejects H₀, then one can conclude that S is biased for R. If the hypothesis test does not reject H₀, then one cannot conclude that S is unbiased because the power to reject H₀ may have been low due to limited sample size. Rejection of H₀ does not necessarily mean that the bias of S for R is clinically significant. A multivariate confidence interval on D would be helpful in assessing clinical significance of the bias by comparing the interval with acceptable bias thresholds for each phenotyope category (Supplement).

4.2.6. Symmetry

A classifier and the reference phenotype can have homogeneous marginal proportions, yet asymmetric joint probabilities of classification. Asymmetry is a sign that the classifier is locally biased for some phenotypes. For example, in the liver fibrosis data (Table 2), TE called 19 cases of fibrosis stage F2 as F0–1, but only 3 F0–1 cases as F2. A description of the Bowker’s Pearson X² test for symmetry⁶¹, as well as illustration with the SWE data, is given in the Supplemental Material.

4.2.7. Association and Agreement Models

Association models for categorical variables have parameters such as odd ratios that quantify association between the variables. Logistic and multinomial regression models designate one variable as the outcome and the other variables as predictors. Log linear models are agnostic, i.e., do not designate variables as either outcome or predictor.

Agreement models are less commonly used, but include parameters that quantify agreement between variables. A review of agreement models is Agresti (1992)⁶². In the Supplemental Material, we illustrate various association and agreement models and metrics using the liver fibrosis SWE and TE data.

4.3. Interchangeability

For quantitative measurements, interchangeability assessment is reviewed in the Supplemental Material. For classifiers, interchangeability assessment has received relatively little attention. Obuchowski et al (2014)⁶³ discuss the concept of interchangeability of imaging tests.

When a reference standard for determining the true phenotype is unavailable, the accuracy of a classifier cannot be evaluated. Unfortunately, the literature is replete with examples in which classification accuracy is nonetheless “evaluated” based on agreement with an imperfect reference standard. When the classifier and imperfect reference standard are conditionally independent given the true phenotype, classifier accuracy will tend to be underestimated with this evaluation. However, serious overstatements of accuracy can occur when the classifier and the imperfect reference standard are positively correlated. Common reasons for positive correlation are (1) the classifier was trained to agree with the imperfect reference standard, and (2) the classifier and imperfect reference standard are based on the same or similar measurement procedure(s), that is, examination of the same medium or milieu (e.g., image or biological sample), measurement of the same or similar analyte, application of the same or similar technology by which measurement(s) are taken, and/or application of the same or similar calculation algorithm for obtaining the final classification result. An example of (1) is a machine learning classifier trained to agree with reader interpretations of images that is validated for accuracy in a separate study to classifier agreement with a consensus of reader interpretations. An example of (2) is evaluation of the “accuracy” of a nucleic acid amplification test (NAAT) for chlamydia trachomatis based on agreement with the consensus of two comparator NAAT assays according to the patient infection status algorithm⁶⁴. Pepe (2003) provides numerical examples of the types of bias in diagnostic test accuracy that are introduced depending on the relationship between a test and an imperfect reference standard²².

When a reference standard is unavailable for determining the phenotype, the accuracy of a classifier cannot be estimated. In the absence of a reference standard, a classifier variable S could be evaluated for interchangeability with a clinically accepted comparator classifier T based on their agreement. A classifier is interchangeable with a comparator if one can be swapped for the other with negligible change in clinical benefit. Interchangeability would be a meaningful claim to have for a classifier if it had advantages over the comparator, e.g., less invasive, less time-consuming, less costly, more automated, or more reproducible.

The effect on clinical benefit of swapping S for T should be negligible when (a) S agrees with T as much as T agrees with itself in replicate testing, and (b) the distribution of disagreements between S and T is symmetric, as is assumed to be true for replicate values of T. Criterion (a) is the concept of individual equivalence^65,66. Obuchowski (2001)⁶⁷ evaluated individual equivalence of electronic medical images (e.g., full-field digital mammography) with screen film, comparing the number of cases with ‘equivalent’ reader diagnoses on electronic images and film to the number of cases with ‘equivalent’ diagnoses when the readers are just using film.

Symmetry is related to the concept of exchangeability. S and T are exchangeable if their labels can be permuted without altering their joint distribution. Symmetry of S and T cannot be shown with statistical significance because estimates of cell probabilities are uncertain in finite samples. For hypothesis testing, non-inferiority margin(s) in asymmetry of the cell probabilities could be pre-specified and tested for statistical significance.

We contend that agreement and symmetry are the hallmarks of interchangeability. Agreement is not the same as association. As mentioned, two categorical variables can be highly associated yet agree poorly because their joint distribution is asymmetric. However, strong association together with symmetry implies high agreement. Thus, interchangeability of two classifiers can be assessed by examining their association and symmetry.

5. A Case Study: Atherosclerotic Plaque

In this section we provide a real-world example of a phenotype classification task, taking the example from an understanding of the clinical context to model development, and finally model evaluation. Figure 1 provides a pictorial overview of model development process.

Figure 1:

An overview of the atherosclerotic plaque classification task. A) Standardized CTA images are acquired in accordance with the existing QIBA profile. B) Images are used to classify tissue type and location at each cross section along the vessel wall. Vessel morphology is validated against histological truth. C) Cross sectional views along the vessel depict tissue type and location as color overlays to the vessel wall. D) Area and location of tissue at each cross section is used as input into a convolutional neural network to generate a phenotype classification (subclinical, stable, unstable) at each vessel cross section. In this way, variations in raw images are filtered out, and focus remains on validated tissue characteristics. Phenotype classification is compared against agreement with an expert pathologist blinded to imaging results.

5.1. Clinical Context

Despite advances in therapy, reduction in myocardial infarction or death remains elusive. Cardiovascular disease is the most common cause of death and disability in the world, mainly by myocardial infarction and ischemic stroke resulting from unstable atherosclerosis⁶⁸. A clear need exists for more precise patient categorization. Despite discoveries of new predictive plasma biomarkers⁶⁹, routine diagnostic methods for identification of individuals and lesions at high risk for atherothrombosis in coronary or extracranial arteries are still lacking⁷⁰.

5.2. Classification Task

In this example, we aim to categorize plaque stability phenotype from computed tomography angiography (CTA) images. The plaque phenotype classification system approved by the American Heart Association (AHA) to inform practicing clinicians on the current presentation of the plaque is tied to levels of severity regarding propensity to rupture. Stary’s initial system⁷¹ has been updated and refine in recent years^72,73. For this application, the granularity of the AHA system was reduced to strike an optimal balance between feature visibility in radiology and the clinical utility of the classification. We aimed to maximize the value of in vivo radiology relative to clinical utility by identifying what a pathologist would say were they to have the tissue at microscopy, but doing so without tissue. From a mathematical point of view, phenotype classification outputs are polychotomous multiclass outcomes (minimal disease, stable plaque, and unstable plaque) which are not constrained physiologically for a certain order of progression beyond the fact that all disease starts as minimal.

5.3. Model Development

The task is classification of atherosclerotic plaques as minimal disease, stable plaque, or unstable plaque. The labelling of plaque type at microscopy sections paired with computed tomography angiography (CTA) serves as the reference standard. Specifically, CTA tissue classification (validated against microscopy sections with outlined tissue type) is the validated model input (Figure 1A-C), while correctness of classification (that is, model output) is assessed relative to a consensus panel of pathologists. Both are blinded to imaging results.

When deep neural networks are trained, the initial layers learn to detect general shapes, edges, corners, etc. The deeper layers learn fine-grained features of the data. While using transfer learning the initial layers are frozen as to not re-learn the task of general shape/edge/corner detection, but the later layers are unfrozen so that the model can update its learning with the current data. The number of frozen layers is set as a hyperparameter. Given the clinical implications of certain types of misclassifications, a non-constant cost function is most appropriate.

5.4. Model Inputs

CTA has been identified as the most effective tool for assessing ischemic heart disease in the prospective Evaluation of Integrated Cardiac Imaging in Ischemic Heart Disease (EVINCI) comparison⁷⁴. Trials such as the Scottish Computed Tomography of the HEART Trial (SCOT-HEART)⁷⁵, and ISCHEMIA⁷⁶ are increasingly showing the sensitivity of the modality. Although qualitative CTA remains the prevalent clinical modality, quantitative CTA analysis offers great potential for biomarker identification and development. Moreover, there is increasing evidence linking quantitative CTA measurements to molecular phenotype⁷⁷. In this example, quantitative measurements are made with ElucidVivo (Boston, MA) software⁷⁸.

This example benefits from knowledge of the associated disease process. Specifically, it is well known that if atherogenic factors dominate over atheroprotective factors, plaques grow in size, and three basic tissue types start to dominate: extra-cellular matrix (ECM) and/or fibrosis may dominate; vascular smooth muscle cells may become osteogenic which in turn promotes calcification, first as micro calcification and later as macro-calcification; or lipid-rich necrotic core (LRNC) and intra-plaque hemorrhage (IPH) may dominate. Plaques largely composed of ECM and/or fibrosis or macro-calcification are generally considered stable; micro-calcification and/or LRNC/IPH are generally considered to be unstable.

QIBA has published a profile for quantitative biomarkers in atherosclerosis⁷⁹. Quantitative measurements included in the profile include maximum wall thickness, lumen area, wall area, plaque burden, lipid-rich necrotic core area, and calcified area. The QIBA profile outlines procedures for taking measurements within a prescribed anatomical target comprising of one or more vessels, and at perpendicular cross-sections along the centerline of each vessel (Figure 1B). Each cross-section thereby presents as a roughly circular lumen area (representing the blood channel) and an annular wall area (presenting the vessel wall, including plaque with its constituent tissues). These quantitative CTA measurements are used as the basis for input to the classification model. However, we expand upon the scalar measurements outlined in the QIBA profile to include not only magnitude of tissue composition but also location and relationship between tissues at a given location. Thus, quantitative information about the location and type of tissue, rather than the raw CTA images, are used as model inputs (Figure 1C). This approach was chosen rather than a combination of scalar QIBs (such as lumen area, wall thickness, calcified area, etc.) because not only the amounts of tissues, but their spatial relationship, is significant. Additionally, an understanding of the underlying pathological processes allowed focus to remain only on information relevant to that process. Characterization of the tissues individually constitutes a single measurand scenario where identified tissues are QIBs with reference truth by histology⁷⁹.

The phenotype is determined from a combination of the individual validated single QIBs from the QIBA CTA profile, not by any single one (Figure 1D). The raw CTA images themselves are not used as model inputs, as raw images are not validated, and include excess variability not tied to the real phenotype that can cause overfitting in models. Rather, we use the tissue characterization first, and feed color-coded images that have been normalized with respect to the dimensions into the model.

Candidate methods need to mitigate drawbacks of machine learning to keep the benefit of the non-linearity of the network. One drawback is how relatively few samples can be relied on, given the relative difficulty of obtaining annotated samples in medicine as compared to other applications of computer vision, and also, how to ensure there is a mechanistic rationale based in biology rather than the algorithm being a black box.

5.5. Model Validation

Model outputs will be assessed relative to agreement with an expert pathologist with results pending at time of publication. In accordance with good model development practices, model evaluation is performed on a separate test dataset from that used to develop and train the model. Correctness of classification by the developed model is assessed relative to a consensus panel of pathologists blinded to imaging results. Agreement of classification by the developed model (i.e., classifying as v=1 (minimal), 2 (stable), and 3 (unstable)) with the corresponding truth states (t=1 (minimal), 2 (stable), and 3 (unstable)) is calculated via the weighted kappa statistic that indicates the proportion of agreement between the assessment categories of two classification results, beyond that expected by chance, penalizing disagreements in terms of their seriousness. The weighted kappa, quantified as κ = (p_o − p_e)/(1 − p_e), compares the observed agreement $p_o={\sum }_{t=1}^3{\sum }_{v=1}^3{w}_{tv}{p}_{tv}$ with the chance expected agreement $p_e={\sum }_{t=1}^3{\sum }_{v=1}^3{w}_{tv}{p}_{t•}{p}_{•v}$, where p_tv, p_t•, p_•v are the joint and the marginal proportions of the corresponding cross-classification table⁸⁰. The weights located at the diagonal represent agreement and thus equal 1. Off diagonal weights indicate the seriousness of disagreement and for the quadratic weighting scheme are given by w_tv = 1 − |t − v|²/4.

The supportable claim is the ability of the model to classify atherosclerotic plaques into one of three phenotypes: “minimal” (which includes normal), “stable”, or “unstable” with a high level of agreement relative to expert assessed histopathology. This could be viewed as a cross-section diagnostic biomarker per the NIH BEST definitions.

6. Discussion

We have provided a discussion on the utility of combining multiple quantitative imaging biomarkers to perform a phenotype classification task and proposed methodology through which it is possible to assess the performance of such a classifier and demonstrate accuracy or interchangeability of the classification task. The methods proposed here build upon existing methodology for univariate quantitative imaging biomarkers and attempt to extend those methods to multiparametric classification tasks. We also reviewed statistical methods for developing a model for a phenotype classification task. Finally, we provided a real-world, end-to-end example in which an imaging-based deep neural network was developed and validated to classify atherosclerotic plaques as minimal, stable, or unstable based on CTA imaging.

6.1. Challenges unique to this use case

Accuracy of a measurement is traditionally assessed by comparison to an objectively measurable reference value. In phenotype classification, the clinically established phenotype is accepted as the reference value to which the imaging-derived phenotype is compared. This may present a limitation to the claims that can be made for this phenotype classifiers because for many disease conditions, the phenotype cannot be established unequivocally, as a clinical reference standard for determining the phenotype is often unavailable. For example, majority consensus interpretation of images is sometimes used as the reference in radiological applications. However, majority consensus is typically imperfect and thus not clinical proof of the phenotype⁸¹. Moreover, an algorithm for a phenotype classifier trained to agree with reader image interpretations would result in the classifier and majority consensus being positively correlated. That is, the classifier would tend to agree with misclassification errors made by majority consensus, resulting in classifier accuracy being overstated. Therefore, for classifier accuracy evaluation, we limited our profile development efforts to the subset of phenotypes for which a reference standard/best available method exists and is based upon something other than radiological imaging (e.g., histopathology or genomic expression). It is recognized that radiological imaging may not always be able to capture a phenotype with the same granularity as the reference method; thus sub-classification or grouping of an established phenotype into coarser metrics may be necessary for comparison to the imaging-based method. When a reference standard does not exist and accuracy cannot be evaluated, a classifier may be assessed for interchangeability with an imperfect reference or a commonly used comparator classifier by evaluating the two classification algorithms for association and symmetry. However, more research is needed on interchangeability assessment of classifiers.

Because classifications are not quantitative, the within-subject standard deviation (wSD) cannot be used to characterize the imprecision (measurement variability) of the phenotype classification task. To quantify categorical variation, we considered the Gini index, entropy, and classification error, which are valid for nominal variables but to do not exploit the ordinal nature of ordinal variables.

Model-based phenotype classification is a many-to-one function of input QIBs. This makes quantification of model imprecision even more difficult, as the phenotype class depends on imprecision in QIB input variables and their correlations. Imprecision of phenotype class (its distribution in replicate classification results) can be approximated analytically from imprecision of QIB inputs using the multivariate delta method⁸². The delta method approximation is accurate in large samples but may be poor in small samples and may be prohibitive to compute for complex, non-linear models. Simulation may be necessary. Finally, imprecision of QIB inputs (e.g., wSDs) might not be constant, e.g., might be proportional to the measurand reference value. Thus, imprecision of phenotype class may be larger or smaller depending on the input values that rendered the same output class. Further study is warranted in how best to combine imprecision of individual model inputs to quantify imprecision of the classification model output. While we have focused on the model output being the phenotype assignment, it could also be a vector of probabilities of each phenotype with assignment based on minimizing expected cost, or a continuous score to which thresholds are applied to determine assignment. We recognize that the same score arising from different model inputs may have different imprecision. Whether (and how) information on score imprecision should be retained to help inform clinical decision making likely depends upon the individual use case. Imprecision quantification of phenotype classification may require special considerations and statistical methods.

An area of continued discussion is the needed level of validation for model input variables. While we recognize that fully validated QIBs with clear biological connection are the ideal situation, we stop short in this manuscript of making that unilateral recommendation. Rather we state that model inputs should be well-characterized and stability of model inputs is necessary to inform development of a robust and reliable model. This position is intended to balance model performance against overly burdensome development activities.

6.2. Gaps

Evaluation of phenotype classifiers is a developing and evolving space. The need for biological explain-ability or interpretability of models based on machine learning influences both the technical design of proposed methods and their validation as a phenotype classifier, but is absolutely not unique to this use case⁸³. Especially when classification models are developed using deep learning networks, use of objectively validated inputs rather than raw images coupled with understanding the characteristics of model decisions is important to ensuring confidence in the model.

We have assumed that the phenotype classifications are mutually exclusive and exhaustive. In our example, the METAVIR fibrosis stages are mutually exclusive and exhaustive for the patient population. However, in some clinical settings a subject can simultaneously have more than one phenotype. For example, a head CT scan may indicate presence of one or more conditions such as intracranial hemorrhage, mass effect, midline shift, or cranial fracture⁸⁴. In this setting, many combinations of phenotypes are possible. Evaluating the accuracy of a classifier to correctly identify every particular combination can be a daunting task and may require statistical adjustment for multiplicity or models to smooth estimates.

We assumed data were cross-sectional and thus did not consider disease stage transitions over time. Over time subjects can progress from an early disease stage to a later disease stage. Subjects can also regress back to an earlier stage from a later stage. For example, stage 2 cervical intraepithelial neoplasia, can regress back to normal especially in young women⁸⁵. Multistate models of event history data can model the probabilities of disease transitions over time, which can be helpful for determining an appropriate screening interval. The models can be extended to incorporate misclassification error⁸⁶.

Clinical benefit, also called clinical utility⁸⁷ or patient outcome efficacy⁸⁸, can be defined as the effect a phenotype classifier has on efficacy outcome(s) when it is used in the intended use population per instructions for use. In clinical outcome trials of diagnostic tests for statistical efficiency, the treatments under study are randomly allocated to the subset of subjects in whom the test result would change the treatment decision if the tests were used in practice^89,90. While we did not consider decision analysis, it can be used to quantify the net benefit of a diagnostic test based on the clinical consequences of testing and diagnostic error⁹¹.

In evaluating classifier accuracy, agreement models were considered in the Supplementary Material. These models contain parameters specific for assessing agreement (as opposed to association). As more experience is gained with the parameters of agreement models, they are expected to provide further insights into the evaluation of classifier accuracy and interchangeability.

Phenotype classifiers S and T were considered interchangeable if swapping one for the other in a clinical practice setting would result in a negligible change on clinical benefit to patients. The interchangeability assessments of association and symmetry presented in Section 4.3 and in the supplementary material are novel in QIB evaluation. Association and symmetry were considered for assessing interchangeability because together they imply high agreement and because symmetry is required for the result of one classifier to behave as if it were a repeated measure from the other classifier. More experience is needed before these assessments can be recommended for wide adoption.

Many if not most machine learning models do not provide the uncertainty of model output. For example, a phenotype classification model that reports the probabilities of having the phenotypes typically does not also report confidence intervals on these probabilities. How best to address uncertainty of the classification output remains an open area of consideration. We believe that communicating the uncertainty of a classifier result to the user facilitates proper use of results in clinical decision making.

How to evaluate if model-estimated probabilities of phenotypes are well-calibrated with actual probabilities is discussed in the overview⁷ and Use Case 3 (Huang et al)⁹. Well-calibrated probabilities can be useful for individualized clinical decision support. For example, the probability of a phenotype may be much higher than the probabilities of the other possible phenotypes and may exceed a threshold recommended for taking clinical action. Model-estimated probabilities of phenotypes can be regarded as updates to the prevalences of the phenotypes given the model inputs. If the model was trained on dataset(s) with phenotype prevalences that are distorted relative to those in the population on whom the model will be used, then the model-estimated probabilities will not be well-calibrated to the actual probabilities and will require adjustment before they can be clinically useful in that population

7. Conclusions

We have outlined an approach and statistical methodology for development of a phenotype classification task from a set of multiparametric QIBs. We have outlined performance metrics to consider and the diagnostic accuracy and interchangeability claims we believe are relevant and supportable for a phenotype classification task informed by multiple QIB model inputs. We discuss various approaches to model development and present options for statistical methodology that can be useful in assessment of the developed model. We intend this manuscript helps to further and advance the robust and promising science of multiparametric biomarker development.