DADP: Dynamic abnormality detection and progression for longitudinal knee magnetic resonance images from the Osteoarthritis Initiative

Abstract

Osteoarthritis (OA) is the most common disabling joint disease. Magnetic resonance (MR) imaging has been commonly used to assess knee joint degeneration due to its distinct advantage in detecting morphologic cartilage changes. Although several statistical methods over conventional radiography have been developed to perform quantitative cartilage analyses, little work has been done capturing the development and progression of cartilage lesions (or abnormal regions) and how they naturally progress. There are two major challenges, including (i) the lack of building spatial-temporal correspondences and correlations in cartilage thickness and (ii) the spatio-temporal heterogeneity in abnormal regions. The goal of this work is to propose a dynamic abnormality detection and progression (DADP) framework for quantitative cartilage analysis, while addressing the two challenges. First, spatial correspondences are established on flattened 2D cartilage thickness maps extracted from 3D knee MR images both across time within each subject and across all subjects. Second, a dynamic functional mixed effects model (DFMEM) is proposed to quantify abnormality progression across time points and subjects, while accounting for the spatio-temporal heterogeneity. We systematically evaluate our DADP using simulations and real data from the Osteoarthritis Initiative (OAI). Our results show that DADP not only effectively detects subject-specific dynamic abnormal regions, but also provides population-level statistical disease mapping and subgroup analysis.

1. Introduction

Osteoarthritis (OA) is a common, chronic disease affecting all joint tissues and characterized by cartilage loss. Over 50 million people have doctor-diagnosed arthritis (Hootman et al., 2016), the majority of which is OA, and even this is likely an underestimate (Jafarzadeh and Felson, 2017), leading to health expenditures totaling over $27 billion annually (Gupta et al., 2005). OA is commonly associated with symptoms including swelling, pain, discomfort, locking, and other problems in mobility, which can be severe (Felson et al., 2000). The disease of OA is also characterized by the progressive loss of joint cartilage, which is believed to be the main factor in OA. Different medical imaging modalities have been employed to investigate and quantitatively measure the knee joint cartilage, such as radiography, Magnetic resonance (MR) imaging, ultrasound (US), Computed Tomography (CT), and optical coherence tomography (OCT) (Braun and Gold, 2012; Kijowski et al., 2020). Among all these available modalities, MR imaging is a non-invasive modality with high spatial resolution. Although MR imaging cannot capture the bone architecture as well as CT, it is useful for capturing some characteristics of the joint, including cartilage morphology and biochemical composition (Guermazi et al., 2003; Roemer et al., 2020). Furthermore, MR imaging is very sensitive to full or partialthickness changes of articular cartilage over time (Huang et al., 2015; Karim et al., 2021). Therefore, MR imaging has been commonly used to detect cartilage morphological changes and understand OA progression (Raynauld, 2003; Cicuttini et al., 2005; Conaghan et al., 2011). Fig. 1 shows MR images of the human knee illustrating cartilage loss.

Fig. 1:

MR images of the human knee illustrating cartilage loss. (left) A sagittal slice of a healthy knee. Bones are annotated in blue, femoral cartilage in green and tibial cartilage in orange; (middle) A coronal slice of the same healthy knee; (right) A coronal slice of an OA knee with cartilage loss indicated by the red arrow.

Several large-scale longitudinal multicenter MR imaging databases, such as the Multicenter Osteoarthritis STudy (MOST) (Segal et al., 2013) and the Osteoarthritis Initiative (OAI) (Eckstein et al., 2012), have been acquired to understand the progression of cartilage change and associated risk factors (Felson et al., 2013). Although quantitative MR image-based biomarkers, such as the volume of the articular cartilage (Eckstein et al., 2006; Schaefer et al., 2017), shape information (An et al., 2016; Bowes et al., 2021), and T₂ relaxometry patterns (Pedoia et al., 2017, 2019), show potential for OA diagnosis and treatment planning, little work has been done in assessing the development of cartilage lesions (or abnormal regions), such as soft tissue swelling or small cartilage or bone fragments, and their natural progression, due to several challenges, which will be detailed below. From a clinical perspective, it is also of great interest to delineate the location and progression of abnormal cartilage regions for refining patient treatment.

There are at least two major challenges in existing abnormal region detection approaches. First, spatio-temporal correspondences and correlations in cartilage thickness are required for statistical localized analysis. It is difficult to establish such correspondences due to the small cartilage volume in relation to the remainder of the knee (Huang et al., 2015; Pedoia et al., 2017, 2019; Roach et al., 2021). Second, abnormal regions across both subjects and time points exhibit spatio-temporal heterogeneity. For the spatial heterogeneity, the cartilage loss pattern is only consistent within a small set of the entire population of subjects. For the temporal heterogeneity, over time cartilage damage and cartilage loss have been found from longitudinal OA studies (Crema et al., 2013; Glyn-Jones et al., 2015; Chen et al., 2017). Thus, treating all the subjects across different time points equally and applying standard statistical methods (e.g., voxel-based analysis and subregion-based analysis (Wirth and Eckstein, 2008)) would be problematic.

To address the two challenges above, a Gaussian Hidden Markov Model (GHMM) was proposed in Huang et al. (2015) to simultaneously model the progression and abnormality of cartilage thickness. However, there are some drawbacks in GHMM. First, GHMM ignores the temporal heterogeneity and assumes that the subject-specific abnormal region pattern is time-invariant, which would be sub-optimal in practice, especially in characterizing and predicting individual OA progression. Second, for normal control subjects, GHMM applies a voxel-wise linear mixed model and treats all voxels as independent units, which ignores two key functional features of imaging data, including spatial smoothness and spatial correlation. Third, GHMM does not provide the population level statistical disease mapping and the detection of some potential OA-subgroups based on the cartilage thickness abnormality (Liu et al., 2021).

The aim of this work is to propose a Dynamic Abnormality Detection and Progression (DADP) framework. First, longitudinal knee MR images are processed to derive flattened 2D cartilage thickness maps. Second, the subject-specific dynamic abnormal regions are detected based on the thickness maps. Compared to GHMM and other existing approaches, the main contributions of our DADP framework are summarized as follows:

1. A deep learning based image analysis pipeline is proposed in DADP to extract 2D cartilage thickness maps from 3D knee MR images, in a common coordinate system, establishing correspondences in space and time;

2. A dynamic functional mixed effects model is proposed in DADP to (i) establish the relationship between the cartilage thickness map and a set of covariates of interest, (ii) capture the spatial smoothness and spatio-temporal correlations within the thickness map through functional data analysis tools (Yuan et al., 2014), and (iii) detect the spatio-temporal heterogeneity in subject-specific dynamic abnormal regions through a dynamic conditional random field model (Sutton et al., 2007);

3. A series of formal statistical inference procedures are established in DADP to (i) test whether there is any significant effect of some covariates of interest on the cartilage thickness, (ii) detect population level OA risk on the thickness map through the statistical disease mapping approach, and (iii) conduct subgroup analysis through a non-negative matrix factorization method (Ding et al., 2005);

4. A powerful and user-friendly package is developed in DADP for the entire OAI MR image dataset. Moreover, our package can be straightforwardly applied for analyzing other knee MR image data at scale. The package will be released to the public through GitHub.

2. DADP

Our DADP includes two main components: (i) an image analysis module and (ii) a statistical modeling module. Specifically, in the image analysis module, the 3D knee MR images are preprocessed through a series of image analysis steps to derive flattened 2D cartilage thickness maps in a common coordinate system. Through this common coordinate system, we obtain spatial correspondence across subjects and/or time points. Then, in the statistical modeling module, a dynamic functional mixed effects model is established based on the 2D cartilage thickness maps to characterize the spatio-temporal correlations within the cartilage thickness map, address the spatio-temporal heterogeneity of abnormal regions, and quantify local OA risk at both population-level and subgroup-level. The workflow of our DADP is summarized in Fig. 2. Next, we introduce the detailed steps in the image analysis module and the estimation and inference procedures in the statistical modeling module.

Fig. 2:

The DADP framework includes (i) an image analysis module and (ii) a statistical modeling module. Image analysis module extracts 2D cartilage thickness maps in a common coordinate system from the 3D knee MR images through a series of image analysis steps. Statistical modeling module includes (a) association analysis between cartilage thickness and covariates of interest, (b) individual abnormal regions detection, and (c) multi-level statistical disease mapping.

2.1. Image analysis module

The aim of this image analysis module is to extract spatially aligned 2D thickness maps of femoral/tibial cartilage from the 3D MR images. The whole pipeline consists of four image preprocessing & analysis steps, including ① segmentation & meshing, ② 3D thickness map computation, ③ registration, and ④ thickness map projection. Fig. 3 shows the overall preprocessing pipeline. First, the femoral cartilage (FC) and tibial cartilage (TC) are segmented from 3D MR images with a U-Net-like network (Ronneberger et al., 2015; Xu et al., 2018), where the model is trained with two parallel output layers. Each layer predicts a binary segmentation for FC or TC, respectively. Compared to some existing methods for knee cartilage segmentation (e.g., graph model based approach (Yin et al., 2010), 2D U-Net network (Norman et al., 2018), volumetric convolution neural network (Raj et al., 2018), and multi-agent learning (Tan et al., 2019)), our proposed cascaded model of simple U-Nets can match the performance of a complex U-Net, while providing better efficiency in terms of using fewer parameters and requiring less memory (Xu et al., 2018). Based on the resulting (soft) segmentations, we build triangle meshes for each cartilage using marching cubes (Lorensen and Cline, 1987). Second, we compute the cartilage thickness for every vertex on the triangle meshes. The thickness value at a vertex on the cartilage surface mesh is approximated by the distance between the vertex’s position and its closest point on the opposite side of the surface. Therefore, we divide the cartilage surface mesh into two opposite parts. Some local smoothing is applied to improve incorrect divisions caused by mesh artifacts and imperfect segmentations. Third, we align the cartilage meshes to an atlas space using a deep registration network in order to align each MR image to an atlas image. Here the atlas is built via an unbiased atlas-building approach. Specifically, we consider 60 affine-aligned images with segmentations. The initial atlas is simply the average image of these aligned affine images. We then iterate B-Spline registrations (using NiftyReg and NMI as similarity measures) to the current atlas image 10 times. After each round of registrations, the atlas image is updated as the average of the set of deformed images given the current registration transformations. In this way, the atlas gets successively sharper. After the 10 registration rounds, we warp the segmentations given the current transformations from image space to the atlas image and obtain the constructed atlas and a probabilistic segmentation. Compared to some widely-used public registration methods (e.g., SyN (Avants et al., 2008, 2014), Demons (Vercauteren et al., 2008, 2009), NiftyReg (Modat et al., 2010, 2014), and Voxel-Morph variant (Dalca et al., 2018)), our proposed registration approach can achieve comparable and sometimes better performance for longitudinal registration with a dramatically reduced computation time and with excellent deformation regularity and symmetry (Shen et al., 2019). Specifically, the registration involves two phases: an affine registration as pre-alignment followed by a non-parametric registration to capture local deformations. For the non-parametric registration, we use a vector momentum-parameterized stationary velocity field (vSVF) approach (Niethammer et al., 2019). In particular, we use its deep learning version (Shen et al., 2019). We then transfer the cartilage thicknesses to the atlas cartilage mesh. Fourth, we project the 3D atlas points onto the 2D plane and interpolate thickness values to obtain spatially aligned flattened 2D cartilage thickness maps for the FC and TC respectively.

Fig. 3:

Image analysis module in DADP: ① segmentation & meshing; ② 3D thickness map computation; ③ registration; and ④ thickness map projection.

2.2. Statistical modeling module

2.2.1. Notations & Assumptions

Suppose that we obtain longitudinal imaging data and clinical variables for n unrelated subjects. Let n_i be the total number of observations for the i-th subject for i = 1, …, n. Our images consist of preprocessed 2D cartilage thickness maps calculated from the OAI study, which are registered to a common template $\mathcal{S}\subset {ℝ}^2$ with n_v grid points $\left\{s_1,\dots ,s_{n_v}\right\}$. For the i-th subject at the j-th time point, we observe (i) 2D imaging data, denoted by y_i,j(s), indicating the cartilage thickness at pixel s; (ii) a p_x-dimensional covariate vector of interest, denoted by x_i,j, including the intercept, time-independent covariates (e.g., gender), and time-dependent ones (e.g., age and body mass index (BMI)); and (iii) the Kellgren-Lawrence grade (KLG) (Kellgren and Lawrence, 1957) denoted as g_i,j. The KLG score corresponds to 5 diagnostic outcomes, starting from normal stage (g_i,j = 0), early stage (g_i,j = 1), mild OA (g_i,j = 2), moderate OA (g_i,j = 3), to severe OA (g_i,j = 4). Without loss of generality, we assume that the first n* subjects are OA patients with at least one observation being at OA stage, whereas the rest n − n* subjects are normal controls with all observations being at normal stage. In addition, for the i-th subject at the j-th time point, we assume that $\mathcal{S}$ can be decomposed into the union of normal regions ${\mathcal{R}}_{i,j}^0$ and abnormal regions ${\mathcal{R}}_{i,j}^1$, that is

$$\mathcal{S}={\mathcal{R}}_{i,j}^0\cup {\mathcal{R}}_{i,j}^1\text{ and }{\mathcal{R}}_{i,j}^0\cap {\mathcal{R}}_{i,j}^1=\varnothing .$$ (1)

Here we also assume that: (i) observations at the normal stage are perfectly healthy and do not exhibit any abnormal regions, i.e., ${\mathcal{R}}_{i,j}^1=\varnothing$; (ii) for OA patients, the shape, size, and location of abnormal regions ${\mathcal{R}}_{i,j}^1$ may vary across subjects and time points; and (iii) the abnormal regions satisfy an inclusion property such that ${\mathcal{R}}_{i,j}^1\subset {\mathcal{R}}_{i,j+1}^1$ for j = 1, …, n_i − 1. To further illustrate these assumptions, Fig. 4 shows three different abnormal region patterns. In this example, patients at the normal stage do not have any abnormal regions, while the shape, size, and location of abnormal regions are different across subjects and time points. In particular, once an abnormal region is present it does not disappear and cannot shrinkage at later time points.

Fig. 4:

Three different abnormal region patterns. P1: Normal stage for the first two time points and transition to an early stage for the follow-up two time points. One abnormal region (red) occurs at the third time point and becomes larger at the fourth time point; P2: Early stage at baseline and transition to late stage for the follow-up three time points. The abnormal regions (red) grow in size and number; P3: Early stage at the first three time points and transition to late stage at the fourth time point. One abnormal region (red) grows over time.

2.2.2. DFMEM: dynamic functional mixed effects model

We propose a dynamic functional mixed effects model (DFMEM) to simultaneously detect individual abnormal regions and capture the relationship between cartilage thickness and covariates of interest. The path diagram of DFMEM is presented in Fig. 5. Specifically, DFMEM is defined as

$$y_{i,j}(s)=x_{i,j}^{\top }\beta (s)+b_{i,j}(s)w_{i,j}^{\top }\alpha +{\mathcal{z}}_{i,j}^{\top }{\gamma }_i(s)+{\eta }_{i,j}(s)+{ϵ}_{i,j}(s)$$ (2)

for j = 1, …, n_i and i = 1, …, n. Here β(s) is a p_x-dimensional vector representing the fixed effect at pixel s, w_i,j is a p_w-dimensional vector of covariates (commonly a subset of x_i,j) associated with the effect α occurring within the abnormal regions (i.e., $\left\{b_{i,j}(s),s\in \mathcal{S}\right\}$, b_i,j(s) = 0 if $s\in {\mathcal{R}}_{i,j}^0$ and b_i,j(s) = 1 otherwise), and z_i,j is a p_z-dimensional vector of covariates (commonly a subset of x_i,j) associated with the random effect γ_i(s). In addition, η_i,j(s) is a stochastic individual function and ϵ_i,j(s) is a measurement error. Furthermore, we assume that γ_i(s), {η_i,j(s)}, and {ϵ_i,j(s)} are independent copies of $\mathbb{G}\mathbb{P}\left(0,{\Sigma }_{\gamma }\left(s,s^{\prime }\right)\right)$, $\mathbb{G}\mathbb{P}\left(0,{\sigma }_{\eta }^2\left(s,s^{\prime }\right)\right)$, and $\mathbb{N}\left(0,{\sigma }_{ϵ}^2(s)\right)$, respectively, where $\mathbb{G}\mathbb{P}\left(\mu (s),\Sigma \left(s,s^{\prime }\right)\right)$ denotes a Gaussian process with mean function μ(s) and covariance function Σ(s, s′). Therefore, given $b_i(s)={\left(b_{i,1}(s),\dots ,b_{i,n_i}(s)\right)}^{\top }$, the covariance structure for $y_i(s)={\left(y_{i,1}(s),\dots ,y_{i,n_i}(s)\right)}^{\top }$ can be written as

$${\Sigma }_{y_i}\left(s,s^{\prime }\right)=Z_i^{\top }{\Sigma }_{\gamma }\left(s,s^{\prime }\right)Z_i+\left({\sigma }_{\eta }^2\left(s,s^{\prime }\right)+{\sigma }_{ϵ}^2(s)\mathbf{1}\left(s=s^{\prime }\right)\right)I_{n_i},$$ (3)

where $Z_i=\left({\mathit{z}}_{i,1},\dots ,{\mathit{z}}_{i,n_i}\right)$, 1(·) is an indicator function, and $I_{n_i}$ is an n_i × n_i identity matrix. Detailed derivations of (3) can be found in Appendix A. Model (2) consists of four key components: (i) a fixed effect component $x_{i,j}^{\top }\beta (s)$ to characterize the fixed effect between cartilage thickness and covariates of interest; (ii) a subject-specific random effect component including $z_{i,j}^{\top }{\gamma }_i(s)$ to characterize the within-subject spatio-temporal correlation structure and η_i,j(s) to capture the spatial correlation structure within $\mathcal{S}$; (iii) measurement errors ϵ_i,j(s); and (iv) a disease related random effect component $b_{i,j}(s)w_{i,j}^{\top }\alpha$ to characterize the disease related dynamic effect within abnormal regions.

Fig. 5:

Path diagram of DFMEM. The change of cartilage thickness at pixel s includes four parts in the proposed model, including (i) the common dynamic change $x_{i,j}^{\top }\beta (s)$ in both normal controls and OA subjects, (ii) the change caused by random effects $\left\{{\mathit{z}}_{i,j}^{\top }{\mathit{\gamma }}_i(s),{\eta }_{i,j}(s)\right\}$, (iii) measurement errors ϵ_i,j(s), and (iv) the change $w_{i,j}^{\top }\mathit{\alpha }$ caused by dynamic abnormal regions (red).

Next, we propose a dynamic conditional random field model (DCRFM) (Wang and Ji, 2005; Wang et al., 2006; Sutton et al., 2007; Yin et al., 2009; Deufemia et al., 2014) to describe the subject-specific latent variables $\left\{b_i(s),s\in \mathcal{S}\right\}$. Some assumptions are introduced here. First, it is assumed that b_i(s), γ_i(s), {η_i,j(s)}, and {ϵ_i,j(s)} are mutually independent. Second, it is assumed that the pattern of b_i,j(s) at pixel s for the i-th OA patient at the j-th time point is affected by three possible factors: (i) population factor $b_0\left({\mathcal{D}}_s\right)=\left\{b_0(s),s\in {\mathcal{D}}_s\right\}$: patterns of pixels in ${\mathcal{D}}_s$, including both pixel s and pixels in its neighborhood, at the population level, (ii) spatial factor $b_{i,j}\left({\mathcal{N}}_s\right)=\left\{b_{i,j}(s),s\in {\mathcal{N}}_s\right\}$: patterns of pixels in ${\mathcal{N}}_s$, the neighborhood of pixel s, for the i-th OA patient at the j-th time point, and (iii) temporal factor $b_{i,j-1}\left({\mathcal{M}}_s\right)=\left\{b_{i,j-1}(s),s\in {\mathcal{M}}_s\right\}$: patterns of pixels in ${\mathcal{M}}_s$, including both pixel s and pixels in its neighborhood, for the i-th OA patient at the previous time point. Specifically, the pattern of b_i,j(s) at baseline is assumed to be affected by the population and spatial factors, whereas that at follow-up visits is assumed to be affected by all the three factors. Further illustration of the three factors is presented in Fig. 6.

Fig. 6:

Illustration of the three factors affecting the pattern of pixel s for one patient at the j-th time point. (i) population factor (yellow): patterns of both pixel s and pixels in its neighborhood at the population level; (ii) spatial factor (red): current patterns of the neighborhood of pixel s at the subject level; and (iii) temporal factor (blue): previous patterns of both pixel s and pixels in its neighborhood at the subject level.

In order to model the population level pattern of b₀(s), a Potts model (Green and Richardson, 2002) is considered here:

$$p_0\left(b_0(s)\mid \tau \right)\propto \text{exp}\left\{-\tau U\left(b_0(s),b_0\left({\mathcal{N}}_s\right)\right)\right\},$$ (4)

where $U\left(b_0(s),b_0\left({\mathcal{N}}_s\right)\right)={\sum }_{s^{\prime }\in {\mathcal{N}}_s}1\left(b_0(s)\ne b_0\left(s^{\prime }\right)\right)$ and the turning parameter τ is introduced to encourages spatial smoothness in homogeneous regions. Then, given $b_0\left({\mathcal{D}}_s\right)$ and $b_i\left({\mathcal{N}}_s\right)$, b_i(s) is assumed to follow a DCRFM with the conditional probability mass function, $p\left(b_i(s)\mid b_i\left({\mathcal{N}}_s\right),b_0\left({\mathcal{D}}_s\right),\tau ,\kappa ,\rho \right)$ given by

$$p_1\left(b_{i,1}(s)\mid b_{i,1}\left({\mathcal{N}}_s\right),b_0\left({\mathcal{D}}_s\right),\tau ,\kappa \right)\times \prod _{j=2}^{n_i}{p}_2\left(b_{i,j}(s)\mid b_{i,j-1}\left({\mathcal{M}}_s\right),b_{i,j}\left({\mathcal{N}}_s\right),b_0\left({\mathcal{D}}_s\right),\tau ,\kappa ,\rho \right),$$ (5)

where $p_1\left(b_{i,1}(s)\mid b_{i,1}\left({\mathcal{N}}_s\right),b_0\left({\mathcal{D}}_s\right),\tau ,\kappa \right)$ is the conditional probability mass function of b_i,j(s) at baseline and proportional to

$$\text{exp}\left\{-\tau U\left(b_{i,1}(s),b_{i,1}\left({\mathcal{N}}_s\right)\right)-\kappa U\left(b_{i,1}(s),b_0\left({\mathcal{D}}_s\right)\right)\right\},$$ (6)

in which κ is introduced to control the population factor affecting the subject-specific abnormal region pattern at pixel s. In addition, $p_2\left(b_{i,j}(s)\mid b_{i,j-1}\left({\mathcal{M}}_s\right),b_{i,j}\left({\mathcal{N}}_s\right),b_0\left({\mathcal{D}}_s\right),\tau ,\kappa ,\rho \right)$ is the conditional probability mass function of b_i,j(s) at follow-up visits. Specifically, in order to make sure that the inclusion property of abnormal region is satisfied, b_i,j(s) = 1 with probability 1 when b_i,j−1(s) = 1. Otherwise, the conditional probability mass function is proportional to

$$\text{exp}\{-\tau U\left(b_{i,j}(s),b_{i,j}\left({\mathcal{N}}_s\right)\right)-\kappa U\left(b_{i,j}(s),b_0\left({\mathcal{D}}_s\right)\right)-\rho U\left(b_{i,j}(s),b_{i,j-1}\left({\mathcal{M}}_s\right)\right)\}.$$ (7)

where ρ influences the strength of temporal dependencies. It should be noted that, ${\mathcal{D}}_s$, ${\mathcal{N}}_s$, and ${\mathcal{M}}_s$ may have different sizes. For simplicity, throughout this work we consider ${\mathcal{N}}_s$ is the set of the closest 8 neighbors of pixel s, and ${\mathcal{D}}_s={\mathcal{M}}_s={\mathcal{N}}_s\cup \left\{s\right\}$.

2.2.3. Estimation procedure

Our next task is to detect the individual abnormal regions ${\left\{b_i(s),s\in \mathcal{S}\right\}}_{i=1}^n$, and estimate all unknown parameters decomposed into three parts: (i) θ including $\beta (s),{\Sigma }_{\gamma }\left(s,s^{\prime }\right)$, ${\sigma }_{\eta }^2\left(s,s^{\prime }\right)$, and ${\sigma }_{ϵ}^2(s)$, (ii) α, and (iii) τ, κ, ρ. The whole estimation procedure can be treated as an iterative approach, in which either the detected abnormal regions or estimated parameters in one of the three parts are updated with others fixed. Specifically, at the (t+1)-th iteration, the detected abnormal regions $\left\{{\widehat{b}}_i^{(t)}(s),s\in \mathcal{S}\right\}$ and current estimates ${\widehat{\Theta }}^{(t)}\doteq \left\{{\widehat{\theta }}^{(t)},{\widehat{\alpha }}^{(t)},{\widehat{\tau }}^{(t)},{\widehat{\kappa }}^{(t)},{\widehat{\rho }}^{(t)}\right\}$ are updated until certain criterion is satisfied. The overall estimation procedure is summarized in Algorithm 1. More detailed updating procedure and initialization procedure can be found in Appendix B and Appendix D, respectively.

2.2.4. Inference Procedure

After obtaining all the estimated parameters and detected abnormal regions, we carry out a series of formal statistical inference procedures consisting of three different statistical tools: (i) hypothesis testing on parameters of interest including β(s) and α; (ii) statistical disease mapping; and (iii) subgroup analysis.

Hypothesis testing.

In practice, we are interested in investigating (i) whether there is any significant effect associated with some covariates of interest on the cartilage thickness, and (ii) whether there is any significant effect associated with covariates of interest on the cartilage thickness within the abnormal regions. Both of these two scientific questions can be written as hypothesis testing problems in the following general forms respectively:

$${ℍ}_0(s):C_{\beta }\beta (s)=0\text{ v.s. }{ℍ}_1(s):C_{\beta }\beta (s)\ne 0,$$ (8)

$${ℍ}_0:C_{\alpha }\alpha =0\text{ v.s. }{ℍ}_1:C_{\alpha }\alpha \ne 0,$$ (9)

where C_β is a 1 × p_x vector and C_α is a 1 × p_w vector. For testing problem (8), a sequence of Wald tests can be constructed as follows:

$${\mathit{T}}_{\beta }(s)=C_{\beta }\widehat{\beta }(s){\left[C_{\beta }{\widehat{\Sigma }}_{\beta }(s)C_{\beta }^{\top }\right]}^{-1}{\widehat{\beta }}^{\top }(s)C_{\beta }^{\top },s\in \mathcal{S},$$ (10)

where ${\widehat{\Sigma }}_{\beta }(s)={\left[{\sum }_{i=1}^n{X}_i{\left\{Z_i^{\top }{\widehat{\Sigma }}_{\gamma }(s,s)Z_i+{\widehat{\sigma }}_{\eta }^2(s,s)I_{n_i}\right\}}^{-1}{X}_i^{\top }\right]}^{-1}$ is the estimated covariance matrix of $\widehat{\beta }(s)$. Given the estimate $\widehat{\mathit{\alpha }}$ and the detected abnormal regions ${\left\{{\widehat{b}}_i(s),s\in \mathcal{S}\right\}}_{i=1}^n$, the corresponding p-values can be derived based on the asymptotic properties of the test statistics under ${ℍ}_0$. In particular, the false discovery rate (FDR) adjustment method (Yekutieli and Benjamini, 1999) can be employed to calculate the adjusted p-values for addressing the multiple comparison issue in (8). In addition to the local test statistics, an integral version of Wald test statistic is established to investigate the global effect on the whole thickness map:

$$T_{\beta }={\int }_{\mathcal{S}}{T}_{\beta }(s)ds,$$ (11)

where the wild bootstrap method is considered in calculating the p-values (Zhu et al., 2010, 2011, 2012, 2019). The details of wild bootstrap procedure can be found in Appendix E. For the testing problem in (9), a typical Wald test is also considered here:

$$T_{\alpha }=C_{\alpha }\widehat{\alpha }{\left[C_{\alpha }{\widehat{\Sigma }}_{\alpha }{C}_{\alpha }^{\top }\right]}^{-1}{\widehat{\alpha }}^{\top }{C}_{\alpha }^{\top },$$ (12)

where ${\widehat{\Sigma }}_{\alpha }$ is the estimated covariance matrix of $\widehat{\mathit{\alpha }}$, which can be calculated via the bootstrap method as well (detailed procedure can be found in Appendix F).

Statistical disease mapping.

In addition to the subject-specific abnormal region pattern, we can also derive the statistical disease mapping at the population level. Specifically, the posterior probability that the pixel site belongs to the abnormal region is calculated as below:

$$\frac{\text{exp}\left\{-\widehat{\tau }{n}^{*}U\left(1,{\widehat{b}}_0\left({\mathcal{N}}_s\right)\right)-\widehat{\kappa }{\sum }_{g_{i,j}>0}U\left(1,{\widehat{b}}_{i,j}\left({\mathcal{D}}_s\right)\right)\right\}}{{\sum }_{l=0}^1\text{exp}\left\{-\widehat{\tau }{n}^{*}U\left(l,{\widehat{b}}_0\left({\mathcal{N}}_s\right)\right)-\widehat{\kappa }{\sum }_{g_{i,j}>0}U\left(l,{\widehat{b}}_{i,j}\left({\mathcal{D}}_s\right)\right)\right\}}.$$ (13)

Therefore, the statistical disease mapping is able to locate OA related regions of interest (ROIs) at the population level and helpful in imaging biomarker development and OA early prevention.

Subgroup analysis.

Besides the abnormal region pattern at both the individual and the population levels, one is also interested in utilizing the abnormal region based imaging markers to identify subgroups with spatially distinct cartilage changes over time. Specifically, standard subgroup outcomes for OA progression (e.g., KLG) have low reliability, are difficult to interpret, and respond poorly to change (Schiphof et al., 2011; Felson et al., 2011; Guermazi et al., 2012). In comparison, our detected individual dynamic abnormal regions are localized cartilage thickness based features and highly sensitive to the OA progression (Balamoody et al., 2010; Wirth et al., 2013). Therefore, the abnormal region based subgroup analysis has advantages in providing improved subgroup measures, which is important in specific OA prevention and clinical treatment management. In order to conduct this clustering analysis, we consider the non-negative matrix factorization (NMF) method which has an inherent clustering property (Ding et al., 2005). Specifically, we first construct a n_v × n* matrix Ω, whose i-th column is defined as ${\Omega }_i={\sum }_{j=1}^{n_i}{\mathit{\omega }}_{i,j}1\left(g_{i,j}>0\right)/{\sum }_{j=1}^{n_i}1\left(g_{i,j}>0\right)$. Here ω_i,j is a n_v × 1 vector, whose l-th element is the estimated probability $\mathbb{P}\left(b_{i,j}\left(s_l\right)=1\right)$. By adopting the NMF method, the approximation of matrix Ω is achieved by minimizing the following loss function:

$$\parallel \Omega -MC{\parallel }_F,\text{ subject to }0\le M\le 1,0\le C,$$ (14)

where ‖ · ‖_F is the Frobenius matrix norm, M is an n_v × k matrix, C is a k × n* matrix, and k is the number of clusters. The computed matrix C gives the cluster membership, i.e., the i-th patient belongs to the l-th cluster if the element C_l,i > C_l′,i for all l′ ≠ l, while the computed matrix M gives the subgroup level statistical disease mapping, i.e., the l-th column in M represents the disease mapping of the l-th cluster. In addition, the consensus matrix (Brunet et al., 2004; Kim and Park, 2007) is adopted here to determine the number of clusters for both simulation studies and real data analysis. The consensus matrix Λ is defined as the average connectivity matrix whose entry Λ_i,i′ reflects the probability that subjects i and i′ belong to the same cluster. To measure the dispersion of a consensus matrix, the dispersion coefficient is defined as ${\rho }_{\Lambda }={\sum }_{i=1}^{n^{*}}{\sum }_{i^{\prime }=1}^{n^{*}}4{\left({\Lambda }_{i,i^{\prime }}-0.5\right)}^2$. After obtaining ρ_Λ values for various k, we can determine the number of clusters from the maximal ρ_Λ.

3. Experiments and results

3.1. OAI data description

OAI Data and/or research tools used in the preparation of this manuscript were obtained and analyzed from the controlled access datasets distributed from the Osteoarthritis Initiative (OAI). OAI is a collaborative informatics system created by the National Institute of Mental Health and the National Institute of Arthritis, Musculoskeletal and Skin Diseases (NIAMS) to provide a worldwide resource to quicken the pace of biomarker identification, scientific investigation and OA drug development. Dataset identifier(s): [NIMH Data Archive Collection ID(s) or NIMH Data Archive Digital Object Identifier (DOI)]. If the Research Project involves collaboration with Submitters or NIH staff (as indicated in the DUC), then Recipient will acknowledge Submitters or NIH staff as co-authors, if appropriate, on any presentation, disclosure, or publication.

We consider the longitudinal MR image data from the OAI study, which comprises normal controls and individuals with, or at high risk of, knee OA incidence or progression. The OAI study contains a total of 4,796 men and women at ages 4579 enrolled between 2004 and 2006 (Eckstein et al., 2012). Those subjects consist of a progression sub-cohort (1,389), an incidence sub-cohort (3,285), and a normal control unexposed reference sub-cohort (122) (Peterfy et al., 2008; Nevitt et al., 2006). Further details about OAI can be found on http://www.oai.ucsf.edu/.

For the MR images, four 3.0 Tesla Siemens Trio MRI scanners were dedicated to imaging the OAI participants at baseline, 12, 24, 36, 48, 72 and 96 months. All images have uniform dimension of 384 × 384 × 160 and resolution of 0.36 × 0.36 × 0.7mm³ per voxel. After preprocessing, we have 19, 915 left knee FC/TC thickness maps from 4, 343 subjects and 21, 229 right knee FC/TC thickness maps from 4, 339 subjects. The demographic and KLG information of the OAI dataset used here is presented in Table 1.

Table 1:

Demographic (gender, BMI, and age) and KLG information of patients at baseline and overall visits.

	Left knee		Right knee
	Baseline	Overall	Baseline	Overall
Gender (F/M)	2,517/1,826	-	2,522/1,817	-
BMI (kg/m2)	28.6 ± 4.8	28.3 ± 4.8	28.6 ± 4.8	28.3 ± 4.8
Age (years)	61.1 ± 9.2	63.2 ± 9.2	61.1 ± 9.2	63.3 ± 9.2
KLG=0	1,729	8,339	1,627	8,223
KLG=1	759	3,640	762	3,914
KLG=2	1,111	4,697	1,214	5,404
KLG=3	603	2,491	587	2,889
KLG=4	141	748	149	799
Total	4,343	19,915	4,339	21,229

3.2. Simulation Studies

We examine the finite sample performance of our DADP framework. Here the simulation data was generated based on the preprocessed 2D thickness maps of normal controls in the OAI study. First, we fitted the following model to the left knee FC thickness maps from normal controls,

$$y_{i,j}(s)={\beta }_1(s)+{\beta }_2(s)\times \text{ gender }+{\beta }_3(s)\times \text{ BMI }+{\beta }_4(s)\times \text{log}(\text{ Age })+{\gamma }_1(s)+{\gamma }_2(s)\times \text{BMI}+{\gamma }_3(s)\times \text{log}(\text{ Age })+{\eta }_{i,j}(s)+{ϵ}_{i,j}(s),$$ (15)

where the assumptions on random effects ${\left\{{\gamma }_l(s)\right\}}_{l=1}^3$, individual stochastic term η_i,j(s), and measurement error ϵ_i,j(s) are the same as those in model (2). Here both BMI and logarithm transformed age information are normalized through the Z-transformation. Then, the obtained parameter estimators of $\left\{\beta (s),{\Sigma }_{\gamma }\left(s,s^{\prime }\right),{\sigma }_{\eta }^2\left(s,s^{\prime }\right),{\sigma }_{ϵ}^2(s)\right\}$ were used as the true values in model (2). In addition, the true values of elements in α were independently generated from U(−0.3, 0) and w_i,j = (1, normalized BMI, normalized log(Age))^⊤.

On the other hand, the individual abnormal regions were pre-specified according to three types of abnormal region patterns based on the statistical disease mapping learned from the real data (see Fig. 7). Specifically, we first selected two seed points with high probability in the population level statistical disease mapping. For each observation with abnormal region pattern 1, one abnormal region is randomly generated with its center around seed point 1; for each observation with abnormal region pattern 2, one abnormal region is randomly generated with its center around seed point 2; and for each observation with abnormal region pattern 3, two abnormal regions are randomly generated with their centers around two seed points respectively. In addition, the sizes of the abnormal region at baseline and follow-up visits vary across subjects. Finally, we generated 624 thickness maps from 160 subjects with 100 normal controls and 60 diseased patients equally divided into 3 groups according to their abnormal region patterns with 3 or 4 observations for each subject, and the average signal-to-noise ratio (SNR) is 11.44.

Fig. 7:

Illustrations of three predefined abnormal region patterns (background is annotated in dark blue; normal region in light blue; abnormal region in yellow). Two seed points are pre-selected according to the population level statistical disease mapping learned from the left knee FC 2D thickness maps. Pattern 1: one abnormal region is randomly generated around seed point 1; Pattern 2: one abnormal region is randomly generated around seed point 2; Pattern 3: two abnormal regions are randomly generated from two seed points respectively. Examples are presented for the three predefined abnormal region patterns.

First, we fit DFMEM in our DADP with the simulated data and investigate the estimation accuracy of all the functional coefficients β_l(s), l = 1, 2, 3, 4. The mean absolute error (MAE), defined as ${\sum }_{k=1}^{n_v}\left|{\widehat{\beta }}_l\left(s_k\right)-{\beta }_l\left(s_k\right)\right|/n_v$, is calculated and reported in Table 2. For comparison, we consider the spatial random effects model (SREM, Zhu et al. (2007); Huang et al. (2015)) integrated with DCRFM, and the corresponding MAEs are reported in Table 2 as well. The key difference between SREM and our DFMEM is that the smoothness of both the thickness maps and functional coefficients are considered in DFMEM rather than SREM. According to Table 2, the estimation accuracy based on our DADP outperforms that based on SREM in terms of MAE, showing the power of functional data analysis tools used in our DADP.

Table 2:

Estimation accuracy of functional coefficients ${\left\{{\beta }_l(s)\right\}}_{l=1}^4$ for DADP and SREM in terms of MAE, defined as ${\sum }_{k=1}^{n_v}\left|{\widehat{\beta }}_l\left(s_k\right)-{\beta }_l\left(s_k\right)\right|/n_v$.

Method	β1(s)	β2(s)	β3(s)	β4(s)
SREM	0.0452	0.0682	0.0323	0.0322
DADP	0.0134	0.0079	0.0048	0.0051

Second, we apply our DADP to detect the abnormal regions for each subject at each time point. For three randomly selected abnormal subjects, the detection results are presented in Fig. 8. It can be seen that the abnormal regions for subjects at each time point can be successfully detected, while the detection performance at follow-up visits is better than that at baseline. The possible reason is that the abnormal regions for subjects at baseline are small and the additional effect in the abnormal region is relatively weak. We also apply the K-means clustering method, SREM+K (SREM integrated with the K-means), and GHMM Huang et al. (2015) to the simulated data. Specifically, the K-means clustering method is the initialization procedure in the statistical modeling module of DADP, and GHMM is an integration of SREM with a Potts model. Compared to the alternative approaches, our DADP has three unique features including (i) the smoothness property of thickness maps, (ii) the dynamic pattern of individual abnormal regions, and (iii) the connection between the individual abnormal region pattern and population level one. For comparison, some metrics, including adjusted Rand index (Hubert and Arabie, 1985), Dice similarity coefficient (Carass et al., 2020), Sensitivity, and Specificity, are considered to evaluate the detection performance of all the four methods for each thickness map, and the mean performance metrics across all thickness maps are reported in Table 3. It can be found that, our DADP outperforms the other three methods in terms of all the performance metrics, indicating that all three features in DADP enhance its performance in terms of dynamic abnormal region detection.

Fig. 8:

Detected abnormal regions for four randomly selected subjects including three patients with different abnormal region patterns (background in dark blue; normal region in light blue; abnormal region in yellow).

Table 3:

Detection performance for DADP and three other methods (K-means, GHMM, and SREM+K) in terms of adjusted Rand index (ARI), Dice similarity coefficient (DSC), Sensitivity (SEN), and Specificity (SPE). Their corresponding standard errors are presented in parentheses.

	ARI	DSC	SEN	SPE
K-means	0.46 (0.24)	0.47 (0.19)	0.69 (0.29)	0.70 (0.22)
SREM+K	0.61 (0.19)	0.64 (0.14)	0.77 (0.16)	0.75 (0.15)
GHMM	0.75 (0.22)	0.77 (0.16)	0.88 (0.12)	0.86 (0.10)
DADP	0.80 (0.18)	0.81 (0.10)	0.98 (0.09)	0.95 (0.01)

Next, we investigate the influences of segmentation accuracy on the performance of dynamic abnormal region detection. To mimic the inaccurate segmentation results, we generated the thickness maps from a misspecified model, where the disease related random effect component was constructed as $b_{i,j}(s)\left[w_{i,j}^{\top }\alpha +{\delta }_{i,j}(s)\right]$ and ${\delta }_{i,j}(s)\stackrel{\text{ iid }}{~}N(0,0.01)$. Here δ_i,j(s) is introduced to capture the potential voxel-wised perturbation caused by the inaccurate segmentation results. The detection performance of all the four methods in terms of ARI is reported in Table 4. It can be found that, although the ARI of our DADP decreases compared to that in Table 3 (i.e., the “perfect segmentation” scenario), our DADP still achieves reasonable detection performance and outperforms the other three methods when the model is misspecified, which means that our DADP is not sensitive to the accuracy of cartilage segmentation results.

Table 4:

Detection performance for DADP with model misspecification and three other methods (K-means, GHMM, and SREM+K) in terms of adjusted Rand index (ARI). Their corresponding standard errors are presented in parentheses.

K-means	SREM+K	GHMM	DADP
0.45 (0.28)	0.58 (0.18)	0.71 (0.24)	0.77 (0.20)

Finally, the statistical disease mapping at the population level is presented in Fig. 9. It can be found that the regions around these two selected seed points are much more likely to be affected by the disease, which is in agreement with the simulation mechanism of abnormal regions. In addition, based on the detected abnormal regions, we conduct the subgroup analysis and present the clustering results in Table 5. It can be found that subjects with abnormal region pattern 2 are successfully identified by subgroup 2, whereas all patients with abnormal region pattern 1 are clustered into the other two subgroups. However, the subjects with abnormal region pattern 3 is not well detected due to two reasons. First, pattern 3 has some overlaps with other two patterns in terms of the design mechanism as shown in Fig. 7. Second, although the mechanisms for the generation of the three patterns are not exactly the same, the simulated abnormal region with pattern 3 may be similar to that with other patterns at baseline or some early time points. For example, the simulated disease region with pattern 3 is almost the same as that with pattern 2 at baseline in Fig. 7. Moreover, the corresponding statistical disease mapping at the subgroup level is also presented in Fig. 9, which successfully recovers the predefined abnormal region patterns. Thus, our DADP provides reasonable subgroup analysis in terms of the abnormal region patterns.

Fig. 9:

Statistical disease mapping at both population level and subgroup level.

Table 5:

Subgroup analysis based on DADP: relationship between clustering results and abnormal region patterns.

	Subgroup 1	Subgroup 2	Subgroup 3
Pattern 1	11	0	6
Pattern 2	0	24	0
Pattern 3	7	3	9

3.3. Real data analysis

First, we consider DFMEM (2) with x_i,j = (1, gender, normalized BMI, normalized log(Age))^⊤, z_i,j = (1, normalized BMI, normalized log(Age))^⊤, and w_i,j = z_i,j. Then, we fit the proposed model on the thickness maps extracted from the left knee FC/TC and the right knee FC/TC, respectively. For the left knee FC, the estimates of coefficient functions associated to the covariates including gender, normalized BMI and normalized log(age) are presented in Fig. 10. In order to test how different covariates locally affect the cartilage thickness, the pixel-wise Wald test statistics in (10) are calculated, and the FDR adjusted − log₁₀ p-values across all pixels are shown in Fig. 10. For the other three cartilages, the estimates of coefficient functions and corresponding FDR adjusted − log₁₀ (p)-values can be found in Appendix G. According to the adjusted p-value maps, compared to the gender effect, both BMI and age effects are more significant on both medial and lateral regions in all four different cartilages, which is consistent with findings in the existing literature (Cai et al., 2019). Meanwhile, the global effects of these covariates on thickness maps are also investigated. The global test statistics in (11) along with the p-values are calculated and reported in Table 6. According to the global p-values, the overall BMI effect is found to be more significant than the age effect on both left knee FC and right knee FC, while the overall age effect is more significant than the BMI effect on both left knee TC and right knee TC. In addition to the coefficient functions, the inference results of α are presented in Table 7. Both BMI and age effects are found significant on the abnormal regions for all the four cartilages.

Fig. 10:

Left knee FC: estimates of functional coefficients associated to the covariates including gender, normalized BMI, and normalized log(Age) (top); corresponding FDR adjusted − log₁₀ p-value maps (bottom).

Table 6:

Inference on coefficient functions β(s): global test statistics and p-values (in parentheses) of three covariates including gender, BMI, and log(Age).

Cartilage	Gender	BMI (kg/m2)	log(Age) (years)
Left knee FC	1.02 (0.45)	31.06 (2.3E-3)	11.43 (2.1E-2)
Left knee TC	2.13 (0.17)	20.01 (8.9E-3)	30.12 (9.8E-4)
Right knee FC	0.85 (0.66)	28.41 (3.6E-3)	13.07 (1.0E-2)
Right knee TC	1.69 (0.33)	20.59 (6.2E-3)	31.21 (1.0E-3)

Table 7:

Inference on coefficients α: estimates associated to three covariates including gender, BMI, and log(Age); and p-values (in parentheses).

Cartilage	BMI (kg/m2)	log(Age) (years)
Left knee FC	−0.15 (0.044)	−0.24 (0.012)
Left knee TC	−0.15 (0.046)	−0.26 (0.009)
Right knee FC	−0.18 (0.032)	−0.30 (0.004)
Right knee TC	−0.17 (0.038)	−0.27 (0.009)

Next, the detected abnormal regions on left knee FC for randomly selected two patients are plotted in Fig. 11. Detection results on other cartilages can be found in Appendix H. The corresponding baseline age, baseline BMI, KLG scores, and time points are also presented. It can be found that the abnormal regions vary across subjects and time points and the inclusion assumption is satisfied in the detection results. Besides the detected individual abnormal regions, the population level statistical disease mapping for different cartilages are calculated and reported in Fig. 12. It can be found that, regions with high probability affected by OA are similar between the left knee and the right knee. This symmetry has also been discovered in the existing studies (Metcalfe et al., 2012).

Fig. 11:

Left knee FC: detected abnormal regions for two randomly selected patients with observations at baseline, 12 months, 24 months, and 48 months: one female (baseline age: 57 years, baseline BMI: 38.6 kg/m², KLG score at baseline is 2, and 3 at follow-up visits) and one male (baseline age: 74 years, baseline BMI: 31.1 kg/m², KLG score is 2 at all four time points).

Fig. 12:

Population level statistical disease mapping for different cartilages: left knee FC (top left), left knee TC (top right), right knee FC (bottom left), and right knee TC (bottom right).

Finally, the subgroup analysis is conducted based on the detected individual abnormal regions. The clustering results are reported in Table 8, in which the KLG information is reported as well to build up the relationship between clustering results and clinical outcomes. According to the clustering results, 78.9% observations at early stage (KLG score 1) are clustered into subgroup 3, 85.5% observations at mild stage (KLG score 2) are clustered into subgroup 1, and 89.0% observations at moderate and severe stages (KLG score 3 or 4) are clustered into subgroups 1 and 2. Similar clustering results can be found on the right knee FC. Specifically, 80.8% observations at early stage (KLG score 1) are clustered into subgroup 3, 84.3% observations at mild stage (KLG score 2) are clustered into subgroup 1, and 73.7% observations at moderate and severe stages (KLG score 3 or 4) are clustered into subgroups 1 and 2. Therefore, the individual abnormal regions on both left knee FC and right knee FC can be treated as useful biomarkers in identifying OA patients at early stage. Moreover, the statistical disease maps at the subgroup level are presented in Fig. 13. It can be found that, for all the different cartilages, each subgroup has its unique statistical disease map, which may be treated as a new imaging biomarker for OA early prevention and treatment development.

Table 8:

Subgroup analysis based on the detected individual abnormal regions: clustering results are summarized according to the KLG scores. Three subgroups are detected for left knee FC and right knee FC, while two subgroups are detected for left knee TC and right knee TC.

Cartilage	Subgroup	KLG
		1	2	3	4
Left knee FC	1	484	3931	713	361
	2	256	326	1449	286
	3	2777	342	179	168
Left knee TC	1	2833	2351	514	405
	2	766	2266	1879	401
Right knee FC	1	597	4594	1220	334
	2	207	441	700	188
	3	3387	415	692	181
Right knee TC	1	2981	2808	701	405
	2	1003	2524	2109	354

Fig. 13:

Subgroup level statistical disease mapping for different cartilages: left knee FC (top left), left knee TC (top right), right knee FC (bottom left), and right knee TC (bottom right).

4. Conclusions & Discussions

In this work, a dynamic abnormal region detection framework, including an image analysis module and a statistical modeling module, has been proposed for quantitative cartilage analysis. In the image analysis module, the spatial correspondence has been well established on 2D cartilage thickness maps extracted from 3D knee MR images through a series of image analysis steps. In the statistical modeling module, a dynamic functional mixed effects model has been proposed. Specifically, the relationship between cartilage thicknesses and covariates of interest has been characterized through functional regression analysis tools, the spatial smoothness and spatial-temporal correlation within the thickness map has been established through functional data analysis tools, and the spatial-temporal heterogeneity in abnormal regions has been detected through dynamic conditional random field models. In addition to the subject-specific dynamic abnormal regions, the population-level statistical disease mapping has been derived, and subgroup analysis has also been conducted through the non-negative matrix factorization method. The proposed framework has been assessed through both simulation studies and real data analysis from the OAI study. It shows that our method outperforms other existing approaches in detecting the subject-specific dynamic abnormal regions and in providing population-level statistical disease mapping and subgroup analysis. All the results derived from our method bring new insights in OA prediction and clinical practice.

Several important issues need to be addressed in our future research. First, the estimation procedure in our DADP can be treated as an iterative approach, and the performance strongly depends on the initialization of abnormal regions and parameter estimation. “Bad” initial values could make the whole algorithm converge to some local maxima or even collapse. To address this issue, a Bayesian framework (Jin et al., 2016; Guo et al., 2020) can be considered here instead, where the estimated parameters can be derived from the posterior distribution via introducing proper prior information, and the latent abnormal regions can be detected via the Monte Carlo Markov Chain (MCMC) sampling method.

Second, in order to detect the subject-specific dynamic abnormal regions, a DCRFM is adopted in this manuscript. Although it works well in our simulated dataset, the simple one-layer latent structure in DCRFM would be inefficient once the spatio-temporal heterogeneity is more complicated within the imaging dataset. To address this issue, the idea of attention-based deep multiple instance learning (Ilse et al., 2018) can be considered and extended to longitudinal datasets. Specifically, both the individual abnormal regions and statistical disease mapping at the subgroup level can be learned through the estimated attention weights.

Third, our current subgroup analysis is conducted via applying the NMF method on the average probability that each pixel belongs to the abnormal region across time, where the dynamic abnormality pattern could not be perfectly characterized. It is of great interest to develop some cartilage thickness progression based features that can integrate the abnormality information to better capture the dynamic abnormality pattern.

Highlights.

• We propose a DADP framework for longitudinal knee MR image analysis

• A DL pipeline is proposed to extract 2D cartilage thickness maps from 3D images

• A DFMEM is proposed for dynamic abnormality detection and progression

• Statistical inferences are introduced for both association and subgroup analysis

• Statistical disease mapping is conducted at both population and subgroup level

Appendix A. Covariance function ${\Sigma }_{y_i}\left(s,s^{\prime }\right)$

Define ${\eta }_i(s)={\left({\eta }_{i,1}(s),\dots ,{\eta }_{i,n_i}(s)\right)}^{\top }$ and ${\mathit{ϵ}}_i(s)={\left({ϵ}_{i,1}(s),\dots ,{ϵ}_{i,n_i}(s)\right)}^{\top }$. Then the covariance function ${\Sigma }_{y_i}\left(s,s^{\prime }\right)$, s′) given b_i(s) can be derived as

$$Z_i^{\top }Cov\left[{\gamma }_i(s),{\gamma }_i\left(s^{\prime }\right)\right]Z_i+Cov\left[{\eta }_i(s),{\eta }_i\left(s^{\prime }\right)\right]+Cov\left[{\mathit{ϵ}}_i(s),{\mathit{ϵ}}_i\left(s^{\prime }\right)\right].$$ (A.1)

According to the assumptions on γ_i(s), η_i,j(s), and ϵ_i,j(s), we have Cov[γ_i(s), γ_i(s′)] = Σ_γ(s, s′), $Cov\left[{\eta }_i(s),{\eta }_i\left(s^{\prime }\right)\right]={\sigma }_{\eta }^2\left(s,s^{\prime }\right)I_{n_i}$, and $Cov\left[{\mathit{ϵ}}_i(s),{\mathit{ϵ}}_i\left(s^{\prime }\right)\right]={\sigma }_{ϵ}^2(s)1\left(s=s^{\prime }\right)I_{n_i}$. Therefore, the covariance structure of y_i(s) given b_i(s) in (3) holds.

Appendix B. Key steps in estimation procedure

Update ${\widehat{\theta }}^{(t)}$.

Given the current detected abnormal regions and estimated parameters in part (ii), model (2) can be written as

$${\tilde{y}}_{i,j}^a(s)=x_{i,j}^{\top }\beta (s)+z_{i,j}^{\top }{\gamma }_i(s)+{\eta }_{i,j}(s)+{ϵ}_{i,j}(s),$$ (B.1)

where ${\tilde{y}}_{i,j}^a(s)=y_{i,j}(s)-{\widehat{b}}_{i,j}^{(t)}(s)w_{i,j}^{\top }{\widehat{\alpha }}^{(t)}$. Then, model (B.1) can be treated as a spatiotemporal (or functional) linear mixed effects model (Bernal-Rusiel et al., 2013; Yuan et al., 2014; Hyun et al., 2016; Zhu et al., 2019). The updating procedure for θ can be summarized in three key steps: (i) derive an initial estimate of β(s) by the local linear regression method (Fan and Gijbels, 1996), while ignoring the temporal structure among repeated observations from one single subject; (ii) derive the updated estimates, ${\widehat{\Sigma }}_{\gamma }^{(t+1)}\left(s,s^{\prime }\right)$, ${\widehat{\sigma }}_{\eta }^{2^{(t+1)}}\left(s,s^{\prime }\right)$, and ${\widehat{\sigma }}_{\eta }^{2^{(t+1)}}\left(s,s^{\prime }\right)$, respectively, by the local constant method (Ramsay and Silverman, 2006) given the updated estimate ${\widehat{\beta }}^{(t+1)}(s)$; and (iii) refine the estimated ${\widehat{\beta }}^{(t+1)}(s)$ by incorporating the temporal-spatial structure obtained from the estimated covariance functions. The detailed updating procedure can be found in Appendix C.

Update ${\widehat{\alpha }}^{(t)}$.

Let $X_i=\left(x_{i,1},\dots ,x_{i,n_i}\right)$ and ${\widehat{W}}_i\left(s_k\right)=W_i\text{diag}\left({\widehat{b}}_i^{(t)}\left(s_k\right)\right)$, where $W_i=\left(w_{i,1},\dots ,w_{i,n_i}\right)$ and $\text{diag}\left({\widehat{b}}_i^{(t)}\left(s_k\right)\right)$ returns a diagonal matrix with ${\widehat{b}}_i^{(t)}\left(s_k\right)\text{s'}$ on the main diagonal. Then, the estimate of α can be updated via minimizing the weighted least squares function

$$\sum _{i=1}^n\sum _{k=1}^{n_v}{\Vert y_i\left(s_k\right)-X_i^{\top }{\widehat{\beta }}^{(t+1)}\left(s_k\right)-{\widehat{W}}_i^{\top }\left(s_k\right)\alpha \Vert }_{{\widehat{\Sigma }}_{i,s_k}}^2,$$ (B.2)

where ${\widehat{\Sigma }}_{i,s_k}=Z_i^{\top }{\widehat{\Sigma }}_{\gamma }^{(t+1)}\left(s_k,s_k\right)Z_i+{\widehat{\sigma }}_{\eta }^{2^{(t+1)}}\left(s_k,s_k\right)$ and the norm operator $\Vert \alpha {\Vert }_{{\widehat{\Sigma }}_{i,s_k}}$ is defined as $\sqrt{{\alpha }^{\top }{\widehat{\Sigma }}_{i,s_k}^{-1}\alpha }$.

Update ${\widehat{b}}_i^{(t)}(s)$.

Here the detected abnormal region pattern at both individual level and population level are updated based on the MRF-MAP approach, which is an efficient method in latent variable recovering and adopted in many practical applications such as image segmentation (Zhang et al., 2001; Nie et al., 2009; Ahmadvand et al., 2017). Specifically, the population level abnormal region pattern ${\widehat{b}}_0^{(t)}(s)$ is updated via minimizing the following loss function

$$\kappa \sum _{g_{i,j}>0}U\left(b_0(s),{\widehat{b}}_{i,j}^{(t)}\left({\mathcal{D}}_s\right)\right)+\tau n^{*}U\left(b_0(s),{\widehat{b}}_0^{(t)}\left({\mathcal{N}}_s\right)\right),$$ (B.3)

while the individual abnormal region pattern is updated via minimizing the following objective function

$${\mathcal{E}}_i(s)-\text{ln }p\left(b_i(s)\mid {\widehat{b}}_i^{(t)}\left({\mathcal{N}}_s\right),{\widehat{b}}_0^{(t)}\left({\mathcal{D}}_s\right),{\widehat{\tau }}^{(t)},{\widehat{\kappa }}^{(t)},{\widehat{\rho }}^{(t)}\right),$$ (B.4)

where ${\mathcal{E}}_i(s)=\frac{1}{2}{\Vert y_i(s)-X_i^{\top }{\widehat{\beta }}^{(t+1)}(s)-W_i^{\top }\text{diag}\left(b_i(s)\right){\widehat{\alpha }}^{(t+1)}\Vert }_{{\check{\Sigma }}_{i,s}}^2$ and ${\check{\Sigma }}_{i,s}={\widehat{\Sigma }}_{i,s}+{\widehat{\sigma }}_{ϵ}^{2^{(t+1)}}(s)I_{n_i}$. Furthermore, the iterated conditional modes (ICM) algorithm (Besag, 1986; Zhang et al., 2001), which uses a greedy iterative strategy for optimization, is adopted to obtain the optimal solutions to (B.3) and (B.4).

Update ${\widehat{\tau }}^{(t)}$, ${\widehat{\kappa }}^{(t)}$, and ${\widehat{\rho }}^{(t)}$.

Since τ, κ, and ρ are not the primary parameters of interest, we use an approximate but computationally efficient method based on a pseudo-likelihood function (Huang et al., 2015). A key advantage of using the pseudo-likelihood function is that it does not involve the intractable partition function. The pseudo-likelihood at the (t +1)-th iteration, denoted as PL(κ, τ, ρ), is a simple product of the conditional likelihood

$$\prod _{i=1}^{n^{*}}\prod _{s\in \mathcal{S}-\partial \mathcal{S}}\frac{p\left({\widehat{b}}_i^{(t+1)}(s)\mid {\widehat{b}}_i^{(t+1)}\left({\mathcal{N}}_s\right),{\widehat{b}}_0^{(t+1)}\left({\mathcal{D}}_s\right),\tau ,\kappa ,\rho \right)}{{\sum }_{b_i(s)}p\left(b_i(s)\mid {\widehat{b}}_i^{(t+1)}\left({\mathcal{N}}_s\right),{\widehat{b}}_0^{(t+1)}\left({\mathcal{D}}_s\right),\tau ,\kappa ,\rho \right)},$$ (B.5)

where $\partial \mathcal{S}$ denotes the set of points at the boundaries of $\mathcal{S}$. Thus, the estimates $\left\{{\widehat{\tau }}^{(t+1)},{\widehat{\kappa }}^{(t+1)},{\widehat{\rho }}^{(t+1)}\right\}$ can be obtained by maximizing the log-pseudo-likelihood function ln PL(τ, κ, ρ) through certain numerical algorithm, e.g., Newton-Raphson method.

Appendix C. Key steps in updating the estimate of θ

Step I.

We first ignore the temporal structure among repeated observations from one single subject and then derive an initial estimate of β(s). Specifically, based on the local linear regression method (Fan and Gijbels, 1996), the initial estimate is obtained via minimizing the following weighted least squares function

$$\sum _{i,j,k}{\left({\tilde{y}}_{i,j}^a\left(s_k\right)-x_{i,j}^{\top }\left(\beta (s)+\stackrel{.}{\beta }(s)\left(s_k-s\right)\right)\right)}^2{K}_H\left(s_k-s\right),$$ (C.1)

where K_H(s) = |H|⁻¹K(H⁻¹s), K(s) is the Gaussian kernel function, and H is the positive definite bandwidth matrix.

Step II.

Given the updated estimate ${\widehat{\beta }}^{(t+1)}(s)$ from (C.1), we first calculate the residuals

$${\widehat{v}}_i(s)=y_i(s)-X_i^{\top }{\widehat{\beta }}^{(t+1)}(s),i=1,\dots ,n.$$ (C.2)

Then, let ${\sigma }_0^2\left(s,s^{\prime }\right)\doteq {\sigma }_{\eta }^2\left(s,s^{\prime }\right)+{\sigma }_{ϵ}^2(s)1\left(s=s^{\prime }\right)$, and the estimates ${\tilde{\Sigma }}_{\gamma }\left(s,s^{\prime }\right)$ and ${\tilde{\sigma }}_0^2\left(s,s^{\prime }\right)$ can be derived by minimizing the following least squares function:

$$\sum _{i=1}^n{\Vert {\widehat{v}}_i(s){\widehat{v}}_i^{\top }\left(s^{\prime }\right)-Z_i^{\top }{\Sigma }_{\gamma }\left(s,s^{\prime }\right)Z_i-{\sigma }_0^2\left(s,s^{\prime }\right)I_{n_i}\Vert }_F^2.$$ (C.3)

Next, by adopting the local constant method (Ramsay and Silverman, 2006), the updated estimates ${\widehat{\Sigma }}_{\gamma }^{(t+1)}\left(s,s^{\prime }\right)$ and ${\widehat{\sigma }}_{\eta }^{2^{(t+1)}}\left(s,s^{\prime }\right)$ can be, respectively, derived by minimizing the following weighted least squares functions,

$$\sum _{k,k^{\prime }}^{n_v}{\Vert {\tilde{\Sigma }}_{\gamma }\left(s_k,s_{k^{\prime }}\right)-{\Sigma }_{\gamma }\left(s,s^{\prime }\right)\Vert }_F^2{K}_H\left(s_k-s\right)K_H\left(s_{k^{\prime }}-s^{\prime }\right),$$ (C.4)

$$\sum _{k,k^{\prime }}^{n_v}{\Vert {\tilde{\sigma }}_0^2\left(s_k,s_{k^{\prime }}\right)-{\sigma }_{\eta }^2\left(s,s^{\prime }\right)\Vert }_F^2{K}_H\left(s_k-s\right)K_H\left(s_{k^{\prime }}-s^{\prime }\right),$$ (C.5)

and the estimate ${\sigma }_{ϵ}^{2^{(t+1)}}$ can be consequently written as

$${\sigma }_{ϵ}^{2^{(t+1)}}(s)={\left({\tilde{\sigma }}_0^2(s,s)-{\widehat{\sigma }}_{\eta }^{2^{(t+1)}}(s,s)\right)}_{+},$$ (C.6)

where the operator (a)₊ is defined as a · 1(a > 0).

Step III.

Let ${\tilde{y}}_i^a\left(s_k\right)={\left({\tilde{y}}_{i,1}^a\left(s_k\right),\dots ,{\tilde{y}}_{i,n_i}^a\left(s_k\right)\right)}^{\top }$ and $X_i=\left(x_{i,1},\dots ,x_{i,n_i}\right)$. Given the estimated covariance functions, we refine the estimate ${\widehat{\beta }}^{(t+1)}(s)$ by minimizing the updated weighted least squares function:

$$\sum _{i=1}^n\sum _{k=1}^{n_v}{\Vert {\tilde{y}}_i^a\left(s_k\right)-X_i^{\top }\left[\beta (s)+\stackrel{.}{\beta }(s)\left(s_k-s\right)\right]\Vert }_{{\widehat{\Sigma }}_{i,s_k}}^2{K}_H\left(s_k-s\right),$$ (C.7)

where the norm operator $\Vert \alpha {\Vert }_{{\widehat{\Sigma }}_{i,s_k}}$ is defined as $\sqrt{{\alpha }^{\top }{\widehat{\Sigma }}_{i,s_k}^{-1}\alpha }$, and ${\widehat{\Sigma }}_{i,s_k}=Z_i^{\top }{\widehat{\Sigma }}_{\gamma }^{(t+1)}\left(s_k,s_k\right)Z_i+{\widehat{\sigma }}_{\eta }^{2^{(t+1)}}\left(s_k,s_k\right)$. In particular, the bandwidth matrices in (C.1), (C.4), (C.5), and (C.7) can be selected by using a leave-one-curve-out cross-validation method (Zhang and Chen, 2007; Zhu et al., 2019).

Appendix D. Initialization procedure

The initialization of abnormal regions and parameter estimation plays a critical role in our iterative estimation procedure. Here, an initial estimation of θ is first derived based on observations at the normal stage. Then, given the estimated ${\widehat{\theta }}^{(0)}$, an initial detection of abnormal regions ${\left\{{\widehat{b}}_i^{(0)}(s),s\in \mathcal{S}\right\}}_{i=1}^n$ is determined based on the residual maps through a conditional clustering analysis approach. Specifically, the abnormal regions for the i-th OA patient at the disease baseline are initialized via clustering analysis, such as K-means, on the residual map, whereas the abnormal regions at follow-up visits are initialized within the detected abnormal regions at the previous time point. Furthermore, the initial estimate of the population level abnormal region pattern is derived from the specified abnormal regions at disease baseline according to the majority rule. In addition to the initialization of abnormal regions, the initial estimator of α can be derived via minimizing the loss function in (B.2). Finally, for the three tuning parameters {τ, κ, ρ}, we set all of them to 0.5 as initial values, which have been found reasonable according to existing studies (Ahmadvand et al., 2017).

Appendix E. Wild bootstrap procedure for approximating the null distribution of T_β

1. Fit model (2) under the null hypothesis with the original data and retain the fitted values ${\widehat{y}}_{i,j}(s)=x_{i,j}^{\top }\widehat{\beta }(s)+{\widehat{b}}_{i,j}(s)w_{i,j}^{\top }\widehat{\alpha }$, individual functions ${\widehat{f}}_{i.j}(s)=z_{i,j}^{\top }{\widehat{\gamma }}_i(s)+{\widehat{\eta }}_{i,j}(s)$, and residuals ${\widehat{ϵ}}_{i,j}(s)=y_{i,j}(s)-{\widehat{y}}_{i,j}(s)-{\widehat{f}}_{i.j}(s)$;

2. Create synthetic response variables $y_{i,j}^{(l)}(s)={\widehat{y}}_{i,j}(s)+{\zeta }_i^{(l)}{\widehat{f}}_{i.j}(s)+{\psi }_{i,j}^{(l)}(s){\widehat{ϵ}}_i(s)$, where ${\zeta }_i^{(l)}$ and $\left\{{\psi }_{i,j}^{(l)}(s),s\in \mathcal{S}\right\}$ are random variables following standard normal distribution;

3. Given $\left\{{\widehat{b}}_{i,j}(s),s\in \mathcal{S}\right\}$, refit the model using the synthetic response variables and calculate the test statistic $T_{\beta }^{(l)}$;

4. Aggregate the results of Steps 2 and 3 over l = 1, …, K (K = 500) to obtain ${\left\{T_{\beta }^{(l)}\right\}}_{l=1}^K$ and calculate the approximated p-value $p=K^{-1}{\sum }_{l=1}^{K}1\left(T_{\beta }^{(l)}>T_{\beta }\right)$.

Appendix F. Wild bootstrap method in estimating the covariance of $\widehat{\alpha }$

1. Fit model (2) under the null hypothesis with the original data and retain the fitted values ${\widehat{y}}_{i,j}(s)=x_{i,j}^{\top }\widehat{\beta }(s)+{\widehat{b}}_{i,j}(s)w_{i,j}^{\top }\widehat{\alpha }$, individual functions ${\widehat{f}}_{i.j}(s)=z_{i,j}^{\top }{\widehat{\gamma }}_i(s)+{\widehat{\eta }}_{i,j}(s)$, and residuals ${\widehat{ϵ}}_{i,j}(s)=y_{i,j}(s)-{\widehat{y}}_{i,j}(s)-{\widehat{f}}_{i.j}(s)$;

2. Create synthetic response variables $y_{i,j}^{(l)}(s)={\widehat{y}}_{i,j}(s)+{\zeta }_i^{(l)}{\widehat{f}}_{i.j}(s)+{\psi }_{i,j}^{(l)}(s){\widehat{ϵ}}_i(s)$, where ${\zeta }_i^{(l)}$ and $\left\{{\psi }_{i,j}^{(l)}(s),s\in \mathcal{S}\right\}$ are random variables following standard normal distribution;

3. Given $\left\{{\widehat{b}}_{i,j}(s),s\in \mathcal{S}\right\}$, refit the model using the synthetic response variables and derive the estimate ${\widehat{\alpha }}^{(l)}$;

4. Aggregate the results of Steps 2 and 3 over l = 1, …, K (K = 500) to obtain ${\left\{{\widehat{\alpha }}^{(l)}\right\}}_{l=1}^K$, where the empirical covariance is calculated to represent the covariance matrix of $\widehat{\alpha }$.

Appendix G. Inference on coefficient functions for left knee TC and right knee FC/TC

Fig. G.14:

Left knee TC: estimates of functional coefficients associated to the covariates including gender, normalized BMI, and normalized log(age) (top); and their corresponding FDR adjusted − log₁₀ p-value maps (bottom).

Fig. G.15:

Right knee FC: estimates of functional coefficients associated to the covariates including gender, normalized BMI, and normalized log(age) (top); and their corresponding FDR adjusted − log₁₀ p-value maps (bottom).

Fig. G.16:

Right knee TC: estimates of functional coefficients associated to the covariates including gender, normalized BMI, and normalized log(age) (top); and their corresponding FDR adjusted − log₁₀ p-value maps (bottom).

Appendix H. Detected abnormal regions for left knee TC nd right knee FC/TC

Fig. H.17:

Left knee TC: detected abnormal regions for two randomly selected patients with observations at five time points: one male (baseline age: 57 years, baseline BMI: 23.4 kg/m², KLG score is 2 at baseline and 12 months, and 3 at 24 months, 36 months, and 48 months) and one female (baseline age: 52 years, baseline BMI: 29.5 kg/m², KLG score is 1 at all five time points, i.e., baseline, 24 months, 48 months, 72 months, and 96 months).

Fig. H.18:

Right knee FC: detected abnormal regions for two randomly selected patients with observations at four time points: one male (baseline age: 64 years, baseline BMI: 25.1 kg/m², KLG score is 1 at all the time points including baseline, 48 months, 72 months, and 96 months) and one female (baseline age: 66 years, baseline BMI: 32.5 kg/m², KLG score is 2 at baseline, and 3 at 24 months, 36 months, and 48 months).

Fig. H.19:

Right knee TC: detected abnormal regions for two randomly selected patients with observations at four time points including baseline, 24 months, 36 months, and 48 months: one female (baseline age: 79 years, baseline BMI: 30.5 kg/m², KLG score is 1 at baseline, and 2 at follow-up visits) and one male (baseline age: 73 years, baseline BMI: 21.6 kg/m², KLG score is 2 at all four time points).