Brain foundation models with hypergraph dynamic adapter for brain disease analysis

Abstract

Brain diseases, such as Alzheimer’s disease and brain tumors, present profound challenges due to their complexity and societal impact. Recent advancements in brain foundation models have shown significant promise in addressing a range of brain-related tasks. However, current brain foundation models are limited by task and data homogeneity, restricted generalization beyond segmentation or classification, and inefficient adaptation to diverse clinical tasks. In this work, we propose SAM-Brain3D, a brain-specific foundation model trained on over 66,000 brain image-label pairs across 14 MRI sub-modalities, and Hypergraph Dynamic Adapter (HyDA), a lightweight adapter for efficient and effective downstream adaptation. SAM-Brain3D captures detailed brain-specific anatomical and modality priors for segmenting diverse brain targets and broader downstream tasks. HyDA leverages hypergraphs to fuse complementary multi-modal data and dynamically generate patient-specific convolutional kernels for multi-scale feature fusion and personalized patient-wise adaptation. Together, our framework excels across a broad spectrum of brain disease segmentation and classification tasks. Extensive experiments demonstrate that our method consistently outperforms existing state-of-the-art approaches, offering a new paradigm for brain disease analysis through multi-modal, multi-scale, and dynamic foundation modeling.

1. Introduction

Brain-related diseases-such as Alzheimer’s disease and brain tumors-constitute some of the most significant challenges in contemporary medicine due to their profound individual and societal impacts [1,2]. Nevertheless, the process of understanding and diagnosing these brain-related diseases is intrinsically complex, primarily due to the need to integrate highly heterogeneous data. Such data span both structural and functional neuroimaging modalities (e.g., MRI and PET), in addition to non-imaging clinical information such as genetic profiles, demographic data, and cognitive scores.

In recent years, artificial intelligence (AI) has become increasingly instrumental in facilitating disease diagnosis and prognosis, largely through its ability to process and analyze large-scale multi-modal medical data [3,4]. To leverage these large-scale data, the medical AI community has begun to develop multi-modal foundation models tailored specifically to clinical domains [3,5,6]. These foundation models have transformed this landscape by delivering impressive performance and generalizability across diverse medical tasks. Among them, models based on the Segment Anything Model (SAM)-such as MedSAM [3], SAM-Med2D [7], and SAM-Med3D [8]-have achieved remarkable success in medical segmentation tasks.

Despite these advances, existing SAM-based models are largely confined to segmentation and do not generalize effectively to other types of tasks. Moreover, general-purpose medical foundation models like SAM are typically trained on datasets involving non-neurological organs, which limits their ability to address brain-specific challenges. To overcome these limitations, several brain-focused foundation models have recently been introduced [9,10]. These models are specialized for brain-related tasks. However, they still suffer from task and data homogeneity: most are designed for a single type of tasks, e.g., either segmentation or classification [6], and for handling a specific data modality, e.g., either EEG [11] or MRI [2,6]. This homogeneity restricts their versatility and broader applicability.

Another major limitation of current brain foundation models is that they lack a powerful mechanism for efficient and effective adaptation to diverse downstream tasks. Some models adopt a cumbersome dual-stage pre-training followed by a third stage fine-tuning [2], thus less efficient. Others either fail to handle both imaging and non-imaging modalities [12], or fall short in simultaneously achieving multi-scale fusion (i.e., fusing low-level morphological and high-level semantic features) and personalized patient-specific adaptation [13]. These shortcomings significantly limit their performance, particularly given the need for complementary multi-modal information, multi-scale morphological and semantic features, and personalized patient analysis in brain disease diagnosis.

In this work, we present a new brain foundation model, SAM-Brain3D, and a Hypergraph Dynamic Adapter (HyDA) to address the above challenges. SAM-Brain3D is trained on 9 brain segmentation datasets, encompassing 14 MRI sub-modalities and 66,280 image-label pairs from 4451 cases. This enables the model to capture detailed anatomical and modality-specific characteristics of the brain for segmenting diverse brain targets, further providing robust prior knowledge for diverse downstream tasks. HyDA is a lightweight adapter designed to efficiently and effectively adapt SAM-Brain3D to downstream brain disease analysis. HyDA leverages hypergraphs to extract complementary information from multi-modal data. It further exploits the high-level semantic features from the hypergraph to dynamically generate convolutional kernels for each patient, with the semantic kernels used to convolve with low-level features for multi-scale feature fusion. The dynamic patient-specific kernels further facilitate personalized adaptation for each individual patient. With SAM-Brain3D and HyDA, our method can excel in diverse clinical tasks, including both segmentation and classification tasks. Our main contributions are summarized as follows:

• Brain Foundation Model for Diverse Tasks: Unlike prior models that predominantly focus on a single task type, our SAM-Brain3D not only segments brain tumor, meningioma, up to 35 brain structures, and broader unseen targets, but also adapts to diverse downstream brain disease classification tasks, promoting strong generalization and transferability.

• Multi-modal, Multi-scale, and Dynamic Adaptation: We propose a novel Hypergraph Dynamic Adapter (HyDA), enabling the encoder of SAM-Brain3D to adapt efficiently to heterogeneous input modalities via multi-scale and patient-specific dynamic adaptation across different tasks.

• Superior Downstream Performance: Our model consistently surpasses current state-of-the-art methods across a variety of brain-related downstream tasks.

The novelty of our work lies not in isolated components, but in their purposeful integration and adaptation to the unique challenges of brain disease analysis. Specifically, we develop 1) a brain-specific foundation model leveraging large-scale MRI data, 2) a lightweight Hypergraph Dynamic Adapter (HyDA) that enables multimodal, multi-scale, and patient-specific adaptation, and 3) a unified framework that supports both segmentation and classification across modalities. Together, these design choices advance beyond prior models in versatility, generalization, and clinical relevance.

2. Related work

Brain Foundation Models

Brain foundation models have emerged as a promising direction in AI for brain disease analysis. These models aim to learn comprehensive and transferable representations of brain anatomy and function from neuroimaging data. Some works have focused exclusively on brain MRI to develop specialized foundation models [2,10,12] without adapting them to other modalities like PET and non-images. In parallel, several studies focus solely on functional modalities such as fMRI and EEG to develop brain foundation models for capturing dynamic neural activity patterns [6,11]. In this work, we introduce a brain foundation model with a novel Hypergraph Dynamic Adapter (HyDA) to support a broader range of clinical and predictive tasks, e.g., brain segmentation and disease diagnosis. Our method can tackle both imaging and non-imaging modalities and facilitate multi-scale feature fusion using a dynamic convolution for better personalized patient-specific disease analysis.

SAM for Medical Tasks

Vision foundation models have achieved remarkable success in general-purpose visual understanding. Among them, the Segment Anything Model (SAM) [14] stands out for its zero-shot and promptable segmentation capabilities. Following its success on natural images, researchers have begun exploring SAM’s applicability to medical imaging. Several studies have demonstrated its effectiveness on 2D medical image segmentation across various modalities [3,7], such as MRI and CT scans. Recently, Wang et al. show that training a fully 3D SAM architecture from scratch on a large-scale medical dataset leads to superior segmentation performance and better generalization [8]. However, existing SAM-based medical foundation models remain predominantly focused on segmentation tasks, with limited extension to other predictive tasks such as classification. This narrow focus restricts their versatility and applicability. Moreover, SAM-based models are typically designed for general-purpose medical imaging tasks and lack specialization for brain-specific applications. In this work, we propose a brain foundation model built upon a 3D SAM framework and trained on a large-scale multi-modal dataset encompassing both structural and functional imaging. Equipped with a Hypergraph Dynamic Adapter, our method can flexibly support various downstream tasks, including both segmentation and classification. Furthermore, it can accommodate tasks involving non-imaging clinical data including genetic profiles, demographic factors, and cognitive assessments.

Hypergraph Neural Networks

Hypergraph Neural Networks (HNNs) [15] have emerged as powerful tools for modeling complex, high-order relations in data. Unlike Graph Neural Networks (GNNs), which model pairwise relations, HNNs represent interactions among multiple nodes through hyperedges. This structure enables HNNs to capture more expressive and structurally rich dependencies [16], making HNNs particularly well-suited for learning from multi-modal datasets where entities are often interrelated in intricate ways [17,18]. In multi-modal learning, HNNs have demonstrated strong capabilities in integrating diverse data types. For example, Yang et al. [18] introduce a hyper-complex graph neural network that enables deep coupling across modalities through cross-embedding and aggregation. Inspired by these advances, we incorporate an HNN-based adapter, HyDA, into our brain foundation model to enhance its adaptability across tasks. HyDA allows the encoder to dynamically adjust to different data distributions including MRI, PET, and non-imaging clinical data, thereby supporting a broad range of downstream neuroimaging and clinical prediction tasks. By leveraging the expressive capacity of HNNs, our model enhances generalization and achieves superior task-specific performance across diverse clinical scenarios.

Alzheimer’s Progression

Alzheimer’s disease (AD) can lead to a decline in brain functions, such as memory loss and cognitive impairment. Alzheimer’s progression aims to detect AD at the mild symptomatic or asymptomatic stage so that the early treatment can be enforced to slow down the progression of AD [19]. Recent studies [20,21] have focused on leveraging multi-modal data to extract complementary information for better Alzheimer’s progression. Multi-modal CNN [20] simply concatenates features of MRI and PET modalities for multi-modal fusion. ProAuto-noAge [21] resorts to auxiliary data of the structural MRI information of AD and Normal Control in addition to PET scans. Multi-scale graph-based grading [22] uses different graphs to process different anatomical scales (e.g., brain structures and hippocampal subfields), with multi-scale features fused for final prediction. It also enforces a consistent predictions from MRI and PET modalities to exploit multi-modal information. HAN [23] employs hypergraph followed with attention to fuse MRI and Morphology modalities. VAP-Former [24] leverages prompt fine-tuning to adapt a pre-trained transformer to MCI progression prediction task and utilizes a transformer block to integrate imaging and non-imaging modalities. The latest MMSDL [25] from 2025 adopts different backbones to process MRI and PET images of different resolutions/scales which are fused via cross-attention. Our method differs from them in two folds. 1) We introduce a brain foundation model for segmenting diverse targets in brain images; 2) We propose a Hypergraph Dynamic Adapter using features from hypergraphs to dynamically generate convolutional kernels for convolution-based multi-scale feature fusion.

MGMT Classification

Recent studies have explored the prediction of 6-methylguanine-DNA methyltransferase (MGMT) promoter methylation status in glioblastoma patients using deep learning applied to MRI scans. Early CNN-based models [26,27] report promising accuracies above 80 % on datasets from TCGA and institutional cohorts. Yogananda et al.[28] further achieve over 90 % accuracy using a T2WI-only 3D-DenseUNet. However, large-scale evaluations, such as the Brain Tumor Radiogenomic Classification Challenge [29], reveal the intrinsic difficulty of the task, where top solutions cannot exceed an AUC of 62 %. Later efforts [30] confirm that directly training deep models on MRI data faces challenges in capturing complex imaging biomarkers. Traditional CNNs, while effective locally, are limited by their reliance on fixed receptive fields, restricting their ability to model global contextual information. To address this, graph-based approaches have been proposed. For instance, Hu et al. [31] introduce a vision graph neural network (ViG) that represents MRI data as graphs, enabling the capture of global and irregular structures. This design alleviates the locality limitations of CNNs and improves performance in MGMT methylation prediction. Despite these advances, existing models still heavily rely on task-specific training with limited data, making them prone to overfitting and lacking robustness. Recent developments in foundation models, such as SAM-Med3D, offer a new solution by enabling powerful, generalizable feature extraction across diverse medical imaging tasks. By leveraging pretrained knowledge from large-scale medical datasets, foundation models can provide richer and more transferable representations.

3. Methodology

As shown in Fig. 1, our method has two stages. The first stage trains SAM-Brain3D on 66,280 image-label pairs across 9 brain-related segmentation datasets. The second stage adapts the foundation model to downstream tasks, e.g., brain disease classification, using a novel Hypergraph Dynamic Adapter (HyDA).

Fig. 1. The pipeline of our method.

(a) Training the brain foundation model of SAM-Brain3D. SAM-Brain3D is trained on diverse brain datasets and numerous image-label pairs for flexible segmentation targets. It shares the same network architecture as SAM-Med3D [8] which has an image encoder, a prompt encoder, and a decoder. (b) Downstream adaptation with Hypergraph Dynamic Adapter (HyDA) for brain disease diagnosis. This stage uses the parameter-fixed image encoder of SAM-Brain3D to extract feature maps/embeddings from multi-modal imaging data while for non-imaging data (optional), a simple Multi-Layer Perceptron (MLP) can be used as a backbone for feature embedding. The embeddings are exploited to construct modality-specific sub-hypergraphs, concatenated as the final hypergraph. The feature maps, the embeddings (node features), and the final hypergraph are then input to HyDA to obtain predictions and enhanced features, which are fed into a discriminative classifier to predict the disease types. We further average both predictions as the final one.

3.1. SAM-Brain3D: A brain foundation model

As depicted in Fig. 1(a), the brain foundation model, termed SAM-Brain3D, is trained on 9 brain-related segmentation datasets, encompassing 14 MRI sub-modalities and 66,280 image-label pairs from 4451 cases. It can segment diverse targets to facilitate brain disease analysis, including stroke lesions, glioma tumors, meningioma, metastatic tumors, and 35 brain structures. The data details are summarized in Table 1. Owing to diverse datasets and massive training data, SAM-Brain3D can excel in segmenting both seen and unseen targets, achieving promising performance and manifesting strong generalization ability (see Table 3). SAM-Brain3D can be directly used for brain segmentation tasks without further adaptation.

Table 1.

Data specifications for the training of SAM-Brain3D. #Mod. denotes the number of MRI sub-modalities and #Pairs denotes the number of image-label pairs.

Datasets	#Mod.	#Cases	#Pairs	Segmentation Targets
Learn2Reg2022 OASIS [32]	1	414	14,486	35 Brain Structures
BraTS2023-MEN	4	1000	8156	3 Meningioma Tumors
BraTS2023-MET	4	238	2528	3 Metastatic Tumors
BraTS2023-GLI	4	1,1611	14,704	3 Adult Glioma Tumors
BraTS2023-SSA	4	43	504	3 African Adult Glioma Tumors
y BraTS2023-PED	4	99	976	3 Pediatric Glioma Tumors
UCSF-PDGM [33]	12	501	23,777	4 Diffuse Glioma Tumors
ATLASR2 [34]	1	655	655	Stroke Lesion
ISLES2022	2	250	494	Stroke Lesion
Total	142	4451	66,280	−

¹The official dataset comprises 1251 cases with labels but 90 cases are randomly chosen for testing.

²The total number of sub-modalities cannot be obtained by simply summing the sub-modalities in each dataset as they may overlap.

Table 3.

Comparison with state-of-the-art foundation models on iSeg19 dataset [47]. iSeg19 is a new, unseen dataset to evaluate the generalization ability of different methods. All the compared methods are based on deep learning. Dice score (%) is reported. CSF: Cerebrospinal Fluid, GM: Gray Matter, WM: White Matter. MACs for slice-by-slice inference methods (SAM-Med2D and MedSAM2) are computed based on an input of N = 64 slices.

Modalities	Methods	#Params	#MACs	CSF	GM	WM	Avg.
MRI-T1	SAM-Med2D	271.53M	5904.16 G	6.81	4.05	4.06	4.97
	MedSAM2	238.96M	1763.83 G	8.79	13.45	12.11	11.45
	SegVol	115.66 M	220.51 G	0.12	1.85	0.55	0.84
	nnInteractive	383.49M	3796.15 G	2.05	3.00	2.10	2.38
	SAM-Med3D-Turbo	100.51 M	89.45 G	3.64	16.85	51.25	23.91
	SAM-Brain3D	100.51 M	89.45 G	15.20	38.47	41.55	31.74
MRI-T2	SAM-Med2D	271.53 M	5904.16 G	6.05	3.71	2.88	4.21
	MedSAM2	238.96 M	1763.83 G	5.69	14.23	15.21	11.71
	SegVol	115.66 M	220.51 G	1.70	0.20	1.10	1.00
	nnInteractive	383.49 M	3796.15 G	2.90	2.05	2.70	2.55
	SAM-Med3D-Turbo	100.51 M	89.45 G	9.86	15.78	20.12	15.25
	SAM-Brain3D	100.51 M	89.45 G	15.79	33.76	36.74	28.76
Average	SAM-Med2D	271.53 M	5904.16 G	6.43	3.88	3.47	4.59
	MedSAM2	238.96 M	1763.83 G	7.24	13.84	13.66	11.58
	SegVol	115.66 M	220.51 G	0.91	1.03	0.83	0.92
	nnInteractive	383.49 M	3796.15 G	2.48	2.53	2.40	2.47
	SAM-Med3D-Turbo	100.51 M	89.45 G	6.75	16.31	35.68	19.58
	SAM-Brain3D	100.51 M	89.45 G	15.49	36.12	39.15	30.25

We build our SAM-Brain3D upon SAM-Med3D [8]. Among the existing foundation models for medical segmentation [3,7], SAM-Med3D is chosen for the following reasons. 1) Since brain imaging data, crucial for brain disease analysis, are usually three-dimensional and multi-modal, SAM-Med3D, a thorough 3D structure trained on massive 3D multi-modal data, can better process these data. 2) As a segmentation foundation model, SAM-Med3D can capture low-level and high-level features from medical images, both of which are important for many downstream asks like Alzheimer’s disease (AD) diagnosis. For instance, both the low-level features (e.g., volumetric changes of gray matter in the hippocampus) and high-level ones, e.g., semantic disease grading information, are necessary for correctly diagnosing AD. 3) SAM-Med3D achieves impressive performance with strong generalization ability. State-of-the-art segmentation performance can underpin better brain structure segmentation while the promising generalizability contributes better to diverse downstream tasks, like brain-related disease diagnosis. Both properties can foster a stronger brain foundation model.

Our SAM-Brain3D shares the same architecture as SAM-Med3D [8], consisting of an image encoder, a prompt encoder, and a decoder. The image encoder extracts feature embeddings from input images. The prompt encoder encodes prompts as embeddings which are provided by users, such as points and masks, to help locate the region of interest. The decoder takes both feature and prompt embeddings to predict the segmentation mask.

To tailor SAM-Brain3D for brain-specific tasks, we first initialize it with the weights from SAM-Med3D-Turbo and train all its parameters using 9 brain-related datasets in Table 1. This approach allowed us to leverage the general medical imaging knowledge from SAM-Med3D while adapting the model to brain-specific segmentation tasks. Details can be found in Section 4.1.

3.2. Hypergraph dynamic adapter for downstream tasks

We propose a novel Hypergraph Dynamic Adapter (HyDA) to adapt SAM-Brain3D to downstream tasks. Since most downstream tasks are about brain disease diagnosis (with or without a specific brain disease), we will focus on classifying brain diseases. Fig. 1(b) depicts the downstream adaptation with HyDA. We use the image encoder of SAM-Brain3D for the feature extraction of imaging modalities and fix its parameters for efficient adaptation. Our framework can also take non-imaging data (optional) as input, such as age, gender, and cognitive scores. The feature embeddings of non-imaging data can be extracted leveraging a simple MLP. The embeddings of both non-imaging and imaging data are utilized to construct modality-specific sub-hypergraphs. These sub-hypergraphs are then fused or concatenated as the final hypergraph. After that, the final hypergraph, the feature maps of imaging modalities, and the feature embeddings of all the modalities will be input to HyDA to obtain the predictions and the enhanced final features. The final features are also fed into a discriminative classifier (e.g., a fully connected layer followed by SoftMax activation function) to predict the disease types. We further average both predictions to make the final diagnosis.

Hypergraph Dynamic Adapter (HyDA) is a lightweight plug-in module designed to have three key features: 1) Multi-modal relation extraction. The HyDA exploits hypergraphs to model high-order relations between different modalities so that complementary information can be extracted for better feature representations. 2) Multi-scale feature fusion. Higher-level information, e.g., semantic global context, will be fused into lower-level features, e.g., texture and shape, as they can be essential for better performance on downstream tasks. 3) Dynamic parameters for subject-wise adaptation. Since the subjects may vary in demographics (e.g., age, gender, and race) and the device for capturing their imaging data may differ in protocols and settings, dynamic parameters, inspired by [35], conditioned on each subject can tackle subject-wise differences and bring better adaptation results. Next, we will elaborate on HyDA.

Hypergraphs for Multi-Modal Data

HyDA leverages hypergraphs to model multi-modal data rather than graphs because hypergraphs alleviate the limitation of graphs (i.e., modeling only pair-wise relation) and excel in modeling high-order relations in multi-modal data. HyDA first constructs hypergraphs using the feature embeddings of different modalities X₁, …, X_M, with M denoting the total modality number. A modality-specific hypergraph G_m is then constructed using $X_m={\left\{x_m^n\right\}}_{n=1}^N$, where $x_m^n$ denotes the features of the nth subject’s mth modality N is the total subject number. G_m has a vertices set V_m, a hyperedge set E_m, and an adjacent matrix H_m, namely G_m = (V_m, E_m, H_m). Each vertex $v_m^n$ in V_m denotes a subject whose features are $x_m^n$. Each hyperedge captures the relations between different vertices/subjects. Unlike an edge in graphs connecting only two vertices, a hyperedge can join more than two vertices, thus capturing the high-order relations among vertices, e.g., their shared properties of brain diseases. In our implementation, each hyperedge $e_m^n$ connects k nearest neighbors center around a given vertex $v_m^n,i.e.,e_m^n=\left\{N{N}_k\left(v_m^n\right)\mid v_m^n\in V_m\right\}$. The hyperedges are concatenated together to obtain the hyperedge set for each modality E_m. The adjacent matrix $H_m\in {ℝ}^{\left|V_m\right|\times \left|E_m\right|}$ defines the weight of each vertex and hyperedge as $h_m\left(v_m,e_m\right)=\{\begin{array}{cc}1,& \text{if}{v}_m\in e_m\\ 0,& \text{otherwise}\end{array}$.

After constructing the modality-specific hypergraphs, we concatenate them as the final hypergraph G = Concat(G₁, …, G_M) to facilitate the extraction of multi-modal information, as shown in Fig. 2. Concretely, we input the final hypergraph and the features of vertices, (G, (X₁, …, X_M)), into spatial hypergraph convolution layers which are presented in [16] and formulated as

$$f=\text{HGConv}\left(G,\left(X_1,\dots ,X_M\right)\right),$$ (1)

where f is the output features of the hypergraph convolution HGConv. The hypergraph convolution aggregates messages via a two-step pass: passing from vertex to hyperedge and from hyperedge to vertex. This two-step strategy [16] can effectively capture the high-order relations across vertices/subjects, so complementary multi-modal information is extracted to comprehensively reflect the brain functions, further producing better predictions of P_g. P_g is obtain using HGConv and a SoftMax activation function ϕ.

$$p_g=\varphi (\text{HGConv}(G,f)).$$ (2)

Fig. 2.

The design of Hypergraph Dynamic Adapter (HyDA) for downstream brain disease diagnosis tasks. HyDA exploits two hypergraph convolution layers to extract high-order relations from the final hypergraph and multi-modal feature embeddings (i.e., node features). It then makes a prediction using a hypergraph classifier. Since the semantic prediction of disease types is based on embeddings, they usually contain semantic global context. The semantic embeddings are input to kernel generators, comprising two 1 × 1×1 convolution layers (‘Conv1’ and ‘Conv2’), to generate semantic and dynamic convolutional kernels for each input subject. The subject-conditioned dynamic kernels then convolve the feature maps of imaging data to fuse the semantic information into the lower-level features, leading to dynamic multi-scale fusion. The fused feature maps are concatenated together and merged using another 1 × 1×1 convolution (‘Conv3’). The merged version will be further enhanced by channel-wise reduction and scale via ‘Conv4&5’. The enhanced feature maps are then flattened into a vector and summed with the original node features via a residual connection as the final feature embeddings. The final embeddings of different modalities are then concatenated and input to a discriminative classifier for prediction.

Multi-scale and Dynamic Feature Fusion

As shown in Fig. 2, we will use the feature embeddings from two hypergraph convolution layers, obtained as per Eq. (1), to enforce multi-scale and dynamic fusion. Let us denote feature embeddings of the nth subject from the first convolution as $f_1^n$ and these from the second as $f_2^n$. Since these two features are high-level features exploited for brain disease prediction, they usually capture semantic global context information. We then integrate their semantic information into lower-level feature maps, which mainly contain morphological or texture information. This multi-scale fusion is achieved by dynamic convolution kernels generated from $f_1^n,f_2^n$, detailed below.

We design kernel generators to generate dynamic kernels from $f_1^n,f_2^n$. Here, ‘dynamic’ means that the parameters of these kernels are not fixed during inference and they are dynamically generated for each specific subject rather than being shared across all the subjects. The dynamic design can help HyDA tackle subject-wise variations and better adapt to each subject for precise and personalized/subject-specific prediction. We will take $f_1^n$ as an example to illustrate the generation of dynamic kernels, but similar operations can be applied to $f_2^n$. Concretely, for a batch of feature embeddings $f_1\in {ℝ}^{B\times C\times 1\times 1\times 1}$ (B as the batch size, C as the channels, and 1 × 1 × 1 denoting the embeddings from a 3D image), the kernel generator first reshapes the feature embeddings of the nth subject $f_1^n\in {ℝ}^{1\times C\times 1\times 1\times 1}$ into ${\widehat{f}}_1^n\in {ℝ}^{1\times C/27\times 3\times 3\times 3}$ using torch.reshape() function. We set the output feature dimension of hypergraphs C to 864 divisible by 27. Then, we increase its channel to C_hid using a 1 × 1×1 convolution layer, i.e., ‘Conv1’ in the top right panel of Fig. 2, leading to the shape of the output feature map being 1 × C_hid × 3 × 3 × 3 where C_hid is the hidden channel dimension. We then swap its first two dimensions using torch.permute() function, leading to C_hid × 1 × 3 × 3 × 3, and apply another convolution ‘Conv2’ to increase its channel to match the desired output channel C_out (C_out = 128 in our case). The output feature maps become the shape of C_hid × C_out × 3 × 3 × 3. Finally, we swap the first two dimensions back to obtain the 3D dynamic kernels $W_1^n\in {ℝ}^{C_{out}\times C_{hid}\times 3\times 3\times 3}$. Formally, the kernel generator is formulated as

$$W_1^n=g_{\theta }\left({\widehat{f}}_1^n\right),$$ (3)

where g_θ denotes the function of kernel generator and θ is the parameter set of these two 1 × 1×1 convolution layers (namely ‘Conv1’ and ‘Conv2’).

The convolutional kernels $W_1^n$ have the following advantages. Firstly, $W_1^n$ are dynamic kernels conditioned on each subject n as different subjects have distinct feature ${\widehat{f}}_1^n$, leading to non-identical kernels of $W_1^n$ for different subject n. Owing to $W_1^n$, HyDA can address subject-wise variations and better adapt to each subject for precise and personalized/subject-specific prediction. Secondly, $W_1^n$ contains semantic information inherited from ${\widehat{f}}_1^n$, enabling better multi-scale fusion via convolving with low-level feature maps:

$$O_{1,m}^n=\sigma \left(W_1^n\otimes F_m^n\right)=\sigma \left(g_{\theta }\left({\widehat{f}}_1^n\right)\otimes F_m^n\right),$$ (4)

Where $O_{1,m}^n$ denotes the output feature map and σ represents the ReLU activation function. ⊗ is the convolution operation while $F_m^n\in {ℝ}^{1\times C_{hid}\times D\times H\times W}$ is the feature map of the nth subject obtained from the image encoder of SAM-Brain3D, with D, H, W denoting depth, height, and width. Eq. (4) enables $W_1^n$ to interact and transform low-level features in a non-linear and subject-adaptive manner, thus more effective in multi-scale fusion compared with simply averaging or concatenating multi-scale features. Since $W_1^n$ contains the semantic global context inherited from ${\widehat{f}}_1^n$ (or $f_1^n$ as the reshape operation does not change the semantic information), the dynamic convolution in Eq. (4) can integrate the semantic global context of $W_1^n$ into morphological information of $F_m^n$ for better brain disease diagnosis.

After obtaining the dynamically generated feature maps $O_{1,m}^n,O_{1,m}^n$ from $f_1^n,f_2^n$, respectively, we concatenate and merge them via convolution layer ‘Conv3’, which results in $O_m^n\in {ℝ}^{1\times C_{hid}\times D\times H\times W}$ sharing the same shape as $F_m^n.O_m^n$ is then enhanced by channel-wise reduction and scale via ‘Conv4’ and ’Conv5’, similar to the Squeeze-and-Excitation (SE) block [36]. The channel reduction ensures that the flattened feature map can match the shape of the original input feature embeddings $x_m^n\in X_m$ to facilitate the element-wise sum. These steps can be formulated as

$$\begin{array}{l}{O}_m^n=g_{{\theta }_3}\left(\text{Concat}\left(O_{1,m}^n,O_{1,m}^n\right)\right),\\ {\widehat{O}}_m^n=\psi \left(g_{{\theta }_4}\left(\text{Pool}\left(O_m^n\right)\right)\right)\odot \sigma \left(g_{{\theta }_5}\left(O_m^n\right)\right),\\ {\tilde{f}}_m^n=x_m^n+\text{Flatten}\left({\widehat{O}}_m^n\right),\end{array}$$ (5)

where $g_{{\theta }_3},g_{{\theta }_4},g_{{\theta }_5}$ denote the ‘Conv3’~’Conv5’, with θ₃, θ₄, θ₅ as their learnable parameters. Ψ stands for the sigmoid activation function while σ for ReLU. ⊙ represents channel-wise scale and Flatten is the operation of reshaping input into a vector. ${\tilde{f}}_m^n$ is the enhanced feature representation to be used for a discriminative classifier for predicting brain diseases.

Remark

Though the kernels $W_1^n$ are conditioned on each subject n, we do not learn a separate set of parameters for every subject. Instead, we design a lightweight kernel generator composed of only two standard 1 × 1×1 convolution layers (denoted as ‘Conv1’ and ‘Conv2’ in Fig. 2) to generate these subject-specific kernels. This generator has only C/27 ∗ C_hid + C_out parameters, making it highly efficient. For instance, in our implementation, C=864, C_hid =384, and C_out = 128, so the total parameters are 864/27×384 + 128≈0.01M. From another perspective, the generated kernels $W_1^n$ can be interpreted as special feature maps output by the kernel generator, dynamically conditioned on each sub-ject. Since kernel generation is essentially a feature map transformation using conventional convolutions, we observe no stability issues during training (as verified in Figure A of the Supplementary Material).

The generated kernels have about 1.27M parameters, with the detailed calculation process as below. After the input feature is reshaped into ${\widehat{f}}_1^n\in {ℝ}^{1\times C/27\times 3\times 3\times 3}$, we use ‘Conv1’ to increase its channel di-mension to C_hid(384 in our implementation), as depicted in the top right panel of Fig. 2. ‘Conv1’ is a convolutional layer with its shape as C/27 × C_hid × 1 × 1 × 1. Its output feature map has the shape of 1 × C_hid × 3 × 3 × 3. We then swap the first two dimensions of the feature map using torch.permute() function, leading to C_hid × 1 × 3 × 3 × 3, and apply another convolution ‘Conv2’ to increase its channel to match the desired output channel C_out (C_out=128 in our case). ‘Conv2’ is a convolutional layer having the shape of 1 × C_out × 1 × 1 × 1. The output feature maps of ‘Conv2’ become the shape of C_hid × C_out × 3 × 3 × 3. Finally, we swap the first two dimensions back to obtain the 3D dynamic kernels $W_1^n\in {ℝ}^{C_{out}\times C_{hid}\times 3\times 3\times 3}$. Therefore, the number of parameters in the genera ted kernels is C out× C_hid × 3 × 3 × 3 = 128 × 384 × 3 × 3 × 3 ≈1.27M.

3.3. Optimization scheme

We have two predictions, P_g from the hypergraph classifier and P_d from the discriminative classifier (comprising a fully connected layer followed by a SoftMax activation function). P_g and P_d are utilized to learn HyDA by optimizing the objective as

$$L=\frac{1}{N}{\displaystyle \sum _{n=1}^N\left[\text{CE}\left(p_g^n,y^n\right)+\text{FL}\left(p_g^n,y^n\right)+\text{CE}\left(p_d^n,y^n\right)+\text{FL}\left(p_d^n,y^n\right)\right],}$$ (6)

where L is the total loss and CE is the cross-entropy loss. FL is focal loss [37] to tackle the class imbalance problem. Then, the average prediction P = (P_g + P_d)/2 is used as the final prediction. Here, we treat the predictions of hypergraph and discriminative classifiers equally because #1 and #2 of Table 4 show that they generally achieve similar performance, particularly in accuracy and Specificity. Thus, it is reasonable to treat them equally can avoid tuning hyperparameter to control their weight.

Table 4.

Verification of key designs on ADNI. Averaged results (%) over 5-fold cross-validation are reported. The last column reports p-values from bootstrap tests comparing each method’s F1 scores with that of HyDA (#4).

#	Methods	ACC	F1	SPE	SEN	p-value
1	Encoder + Discriminative Classifier	85.33	62.27	94.83	55.15	0***
2	Encoder + Hypergraph Classifier	85.33	63.43	94.83	56.58	0***
3	Averaged Prediction of #2 & #3	85.52	65.93	95.17	62.90	0.030*
4	#3 + HyDA	88.09	70.23	96.43	62.12	−

^*p<0.05,

^**p<0.01,

^***p<0.001.

4. Dataset and experimental setup

In this section, we introduce the datasets and experimental setups of training the brain foundation model, SAM-Brain3D, as well as fine-tuning the Hypergraph Dynamic Adapter (HyDA) for two downstream tasks, including Alzheimer’s progression and O6-methylguanine-DNA methyltransferase (MGMT) classification. Our code will be released at https://github.com/Zhongying-Deng/SAM-Brain3D_HyDA/.

4.1. Experimental setup of SAM-Brain3D

We provide the dataset description in the Supplementary Material.

Implementation Details

We set the batch size to 12 and the initial learning rate to 8 × 10⁻⁵, which is 10 % of the learning rate used for SAM-Med3D training. The model is trained for 200 epochs using Adam optimizer. We further reduce the learning rate by 10 at the 120th and 180th epochs.

4.2. Experimental setup of Alzheimer’s progression

Alzheimer’s disease (AD) is a progressive brain disease that causes memory loss, cognitive decline, and even death. The symptoms are usually mild at the early stage, e.g., Mild Cognitive Impairment (MCI) stage, and gradually become more severe over several years. Since MCI patients may convert to AD (progressive MCI) or remain stable (stable MCI), it is important to distinguish these two categories for precise medical intervention. The task of classifying progressive MCI (pMCI) and stable MCI (sMCI) is called the Alzheimer’s progression task. Though AD is not curable, correctly identifying pMCI patients enables early treatment to slow down the progression of converting to AD, thus improving patient outcomes.

Implementation Details

We provide the dataset description for Alzheimer’s progression in the Supplementary Material. We pre-process the 3D MRI volumes as follows: 1) aligning anterior commissure with posterior commissure, 2) removing skull, 3) correcting intensity, 4) removing cerebellum, and 5) applying linear alignment to a template MRI. The PET volumes are aligned with corresponding MRI volumes via linear registration. The MRI and PET volumes are center-cropped into 128 × 128×128 and randomly flipped for data augmentation. Then the images and each feature in non-imaging data are normalized to [0,1].

The images are input into the same image encoder pre-trained in SAM-Brain3D (please see Fig. 1(b)). The non-imaging data are input into a single fully-connected layer to project the original non-image features into the feature space shared by the image features. We fix the image encoder from SAM-Brain3D and train HyDA using AdamW optimizer, with an initial learning rate of 0.001 and batch size of 30. The hypergraphs for HyDA are constructed using k = 20 nearest neighbors by default and k will be evaluated in our experiments. We randomly drop out 50 % vertex features of hypergraphs as a regularization. When optimizing hypergraph convolution layers, we set the weight decay of the AdamW optimizer to 0.01 as another regularization. We run all the experiments on a NVIDIA A100 GPU with 80 GB memory using PyTorch [38].

4.3. Experimental setup of MGMT classification

O⁶-methylguanine-DNA methyltransferase (MGMT) promoter methylation is a key biomarker in glioblastoma, strongly associated with better prognosis and enhanced response to alkylating chemotherapy [39]. Accurate prediction of MGMT status can guide personalized treatment decisions and reduce the need for invasive surgical biopsies. Developing non-invasive imaging-based methods, such as radiogenomic models using MRI, has the potential to accelerate diagnosis, optimize therapy, and improve clinical outcomes for glioblastoma patients [29].

Experimental Settings

We provide the dataset description for MGMT Classification in the Supplementary Material. We adopt a 5-fold cross-validation strategy, with 459 patients for training and 115 for validation in each fold. All models are trained for 25 epochs using the Adam optimizer with an initial learning rate of 0.0001. A learning rate warmup is applied during the first 3 epochs, followed by decay at epoch 6 with a gamma of 0.5. Early stopping with a patience of 7 epochs is employed. Input volumes are resized to 128 × 128 × 128. We use random flipping for data augmentation.

We compare CNN-based models (EfficientNet-B3-3D [40,41], ResNet50-3D [41,42], and the encoder of SegResNet [43]), a transformer-based model (Swin-ViT3D [41,44]), graph-based models (ViG-3D [31] and Population-Graph [45]), and hypergraph-based models. CNN-based and Swin-ViT3D models are trained with a batch size of 8, while graph and hypergraph models are trained with a batch size of 20.

For graph construction, both basic graphs and population graphs are built using k = 8 nearest neighbors, while hypergraphs are constructed with k = 16 nearest neighbors. In CNN-based and Swin-ViT3D models, two fully connected layers with ReLU activation are appended after the encoder. For graph and hypergraph models, two GCNConv layers with ReLU are applied after feature extraction. EfficientNet-B3-3D is trained from scratch due to the lack of publicly available 3D pretrained weights. ResNet50-3D uses Med3D pretrained weights, which are trained on multiple MRI and CT tasks [42]. The encoders of SegResNet and Swin-ViT3D are initialized with pretrained weights from [43] and [46], respectively. All graph and hypergraph models employ the frozen SAM-Brain3D encoder as a feature extractor.

5. Results

This section presents the experimental results of SAM-Brain3D for brain segmentation tasks and HyDA for two downstream brain disease tasks: Alzheimer’s progression and MGMT classification.

5.1. SAM-Brain3D for brain segmentation tasks

We compare SAM-Brain3D with the latest general-purpose medical foundation models on both seen and unseen categories, corresponding to BraTS21 and iSeg19 [47], respectively. These methods include SAM-Med3D, SAM-Med2D [7], MedSAM2 [48], SegVol [49], and nnInteractive [50]. These models are trained on large datasets with diverse modalities and targets, including MRI, CT, and ultrasound but with only a small proportion (<10 %) of brain imaging data. Notably, SAM-Med2D and nnInteractive have seen MRI data (including the training set of BraTS21), while SegVol is trained solely on CT scans and has not encountered any brain MRI during pretraining. We adopt the Dice score (%) to evaluate the segmentation performance.

Table 2 compares SAM-Brain3D with the other competitors on BraTS21 where all its categories are seen during training. We observe a clear better Dice score of SAM-Brain3D over SAM-Med2D, SegVol, nnInteractive, and SAM-Med3D-Turbo on T1, T1ce, and T2 modalities for all three categories. For instance, SAM-Brain3D surpasses SAM-Med3D by 8.63 % when segmenting Edema using T1 MRI, and shows 4.78 % improvement in terms of averaged Dice for T1 modality. SAM-Brain3D is inferior to nnInteractive and SAM-Med3D on the FLAIR modality, probably because FLAIR is less represented in our training set. Despite this, our SAM-Brain3D achieves better average Dice for all four modalities (the last two rows), suggesting its strong potential in brain disease analysis.

Table 2.

Comparison with state-of-the-art foundation models on BraTS21 dataset. All the compared methods are based on deep learning. NE. Tumor: Non-enhancing Tumor, E. Tumor: Enhancing Tumor. The best performance of each modality is in bold.

Modalities	Methods	Edema	NE. Tumor	E. Tumor	Avg.
MRI-T1	SAM-Med2D	15.02	13.19	13.79	14.00
	SegVol	2.10	1.85	1.70	1.88
	nnInteractive	36.90	11.20	40.50	29.53
	SAM-Med3D-Turbo	41.42	15.87	45.09	34.13
	SAM-Brain3D	50.05	17.36	49.31	38.91
MRI-Tlce	SAM-Med2D	13.92	10.64	13.77	12.78
	SegVol	1.54	1.20	1.50	1.41
	nnInteractive	49.60	27.10	59.50	45.40
	SAM-Med3D-Turbo	49.39	26.82	59.31	45.18
	SAM-Brain3D	53.24	29.04	61.37	47.88
MRI-T2	SAM-Med2D	13.10	11.21	12.37	12.23
	SegVol	3.01	2.20	2.75	2.65
	nnInteractive	50.10	17.80	46.20	38.03
	SAM-Med3D-Turbo	53.75	21.34	49.86	41.65
	SAM-Brain3D	58.95	24.26	50.08	44.43
MRI-FLAIR	SAM-Med2D	15.37	14.02	12.98	14.12
	SegVol	0.11	0.05	0.08	0.08
	nnInteractive	61.50	19.50	54.30	45.10
	SAM-Med3D-Turbo	65.45	22.70	58.00	48.72
	SAM-Brain3D	61.93	19.42	50.98	44.11
Average	SAM-Med2D	14.35	12.27	13.23	13.28
	SegVol	1.69	1.33	1.51	1.51
	nnInteractive	49.53	18.90	52.38	40.27
	SAM-Med3D-Turbo	52.50	21.69	53.07	42.42
	SAM-Brain3D	56.04	22.52	52.94	43.83

Table 3 further presents the comparison of segmenting unseen categories from iSeg19. On all three targets and both modalities, SAM-Brain3D demonstrates significant performance improvements, e.g., 13.51 % performance gain over SAM-Med3D for the average Dice on the T2 task. This suggests that SAM-Brain3D benefits from brain knowledge in the 9 brain-related training datasets. This prior knowledge thus effectively enhances the model’s generalization ability in brain-related tasks which is crucial for diverse downstream tasks of brain disease diagnosis. Finally, we observe that SAM-Brain3D contains smaller/comparable number of parameters and MACs (Multiply-Accumulate Operations) than many recent foundation models in medical imaging, demonstrating the efficiency of our SAM-Brain3D.

5.2. HyDA for Alzheimer’s progression

We will first comprehensively evaluate the key designs of HyDA and then compare HyDA with state-of-the-art methods for Alzheimer’s progression. Throughout this section, averaged results (%) over 5-fold cross-validation are reported. We report four metrics: Accuracy (ACC), F1 score, Specificity (SPE), and Sensitivity (SEN).

Effectiveness of Key Designs

We verify the effectiveness of key designs in HyDA by gradually adding each of them to a simple baseline. The baseline freezes the image encoder of SAM-Brain3D and learns a simple discriminative classifier (a fully connected layer followed by Soft-Max activation function). Its results are shown in #1 of Table 4. We then replace the discriminative classifier with a hypergraph-based one, defined as Eq. (2). Compared with #1, #2 improves F1 score by 1.16 % and Sensitivity by 1.43 %, demonstrating the superiority of hypergraph for capturing high-order relations from multi-modal data. When the predictions of these two classifiers are averaged (#3), all four evaluation metrics are increased, particularly 2.50 % improvement of F1 and 6.32 % in terms of Sensitivity. Finally, the multi-scale and dynamic fusion of HyDA (#4) can further boost the accuracy from 85.52 % to 88.09 %, F1 from 65.93 % 70.23 %, and Specificity from 95.17 % to 96.43 %, at the expense of slightly decreasing the Sensitivity. Overall, HyDA (#4) manifests a clear improvement over the baseline of a simple discriminative classifier (#1), which verifies its effectiveness.

Tables 4 reports the results of 5-fold cross-validation, only 5 results usually lead to the limited number of performance samples, constraining the detection of small effect sizes with high confidence. As such, we rerun the 5-fold cross-validation using different random seeds for another two times, leading to 15 results in total for the statistical test. Since 15 performance samples are still limited, we opt for bootstrap tests, which are suitable for limited samples. We further report the p-value of bootstrap tests in Table 4. Here, we choose the F1 score for the statistical tests because it trades off the Specificity and Sensitivity and can better reflect the overall performance in the class imbalance cases than Accuracy. It is also noteworthy that the statistical test operates on aggregated F1/AUC metrics, not subject-level data. We can see that our HyDA beats the other baselines with p-values smaller than 0.05. Thus, the superiority of our method is supported by the statistical significance.

Superiority of SAM-Brain3D’s Image Encoder

Though Table 3 and 2 have justified the superiority of SAM-Brain3D over SAM-Med3D for brain segmentation tasks, we further show in Table 5 that the image encoder of SAM-Brain3D can be better transferred to downstream task for brain disease diagnosis. This is evidenced by the clear decrease in all four metrics when replacing the image encoder of SAM-Brain3D with that of SAM-Med3D.

Table 5. Comparison of SAM-Brain3D with SAM-Med3D for downsteam adaptation on ADNI.

Methods	ACC	F1	SPE	SEN
SAM-Brain3D Encoder + HyDA	88.09	70.23	96.43	62.12
SAM-Med3D Encoder + HyDA	84.24	63.80	92.86	59.57

Evaluation on the Choice of Hypergraphs

By default, we adopt the hypergraph and hypergraph convolution presented in Hypergraph Neural Network Plus (HGNNP) [16], which is a general hypergraph framework. We evaluate this design choice by replacing HGNNP with Dynamic Hypergraph Neural Networks (DHGNN) [51] which updates hypergraph structure using both local and global features. Table 6 shows that HGNNP generally achieves better performance, owing to its general design for handling complicated heterogeneous hypergraphs.

Table 6.

Evaluation on the choice of hypergraphs on ADNI. All the models are trained for 8 h.

Methods	#Params	#MACs	ACC	F1	SPE	SEN
HyDA default (Built on HGNNP)	2.8M	15.2M	88.09	70.23	96.43	62.12
HyDA variant (HyDA-DHGNN)	2.8M	15.2M	87.83	68.21	97.14	59.83
HyDA variant (HyDA-Graphs)	2.8M	15.2M	85.85	66.07	97.14	55.63

We argue that hypergraphs can better model high-order relations in multi-modal data than graphs. To support our argument, we replace hypergraphs of HyDA with graphs and use graph convolution to substitute the hypergraph convolution. We can see in Table 6 that graphs fail to beat their hypergraph counterparts. This comparison underlines the superiority of hypergraphs for multi-modal modeling. We also remark that HyDA is lightweight with only 2.8M parameters and 15.2M MACS (Multiply-Accumulate Operations), efficient for downstream adaption.

Evaluation on the Major Designs of HyDA

Our HyDA has three major designs: hypergraph, multi-scale fusion, dynamic kernels. We evaluate the importance of them on ADNI by discarding each design, respectively. We can see from Table 7 that removing any of these designs can lead to a clear decrease for all the metrics. The degradation presents the benefits of hypergraph, multi-scale fusion, dynamic kernels in HyDA.

Table 7.

Evaluation on the designs of the hypergraph, multi-scale fusion, and dynamic kernels in HyDA. Averaged results (%) over 5-fold cross-validation are reported.

Methods	ACC	F1	SPE	SEN
HyDA (default)	88.09	70.23	96.43	62.12
HyDA w/o hypergraph	82.70	54.3	94.37	50.87
HyDA w/o multi-scale fusion	82.04	54.6	93.57	49.52
HyDA w/o dynamic kernels	81.16	45.3	94.63	39.52

Ablation on Modalities

Our HyDA effectively extracts information from three modalities, i.e., MRI, PET and non-imaging data. We then conduct the ablation on the modalities and show their results in Fig. 3. It is clear that with all three modalities, HyDA achieves the best performance, implying that HyDA can benefit from multi-modal data for better performance. Removing PET results in a clear decrease in both accuracy and F1. We attribute the performance drop to the absence of information in PET, which is essential for HyDA to fuse multi-modal information. It is also worth noting that HyDA can effectively leverage PET for better performance even though our SAM-Brain3D is trained on only MRI data. Further discarding non-imaging modalities and using MRI only can reduce both the accuracy and F1, highlighting the importance of leveraging multi-modal data for good performance.

Fig. 3. Ablation on modalities.

Evaluation on the Number of Nearest Neighbors

When constructing the hypergraphs, we set the number of nearest neighbors k to 20 as the default setting. We further evaluate the sensitivity of the hyperparameter k in Fig. 4. We observe that the accuracy and F1 increase when k varies from 4 to 28 and then decrease with a larger k. The optimal k is 28. Overall, the accuracy is generally stable with the varying k while F1 is more sensitive. Such a difference can be caused by the imbalanced class.

Fig. 4. Evaluation on the nearest neighbor k.

Evaluation on Training Stability

In the “Remark” of Section 3.2, we state that our HyDA does no suffer from stability issues during training. It essentially claims that the model will not collapse during training. To support this claim, we include the loss curve during training in Fig. 5 to show that the loss generally decreases as the training goes on. Thus, our model does not suffer from model collapse.

Fig. 5. Training loss of our model (SAM-Brain3D+HyDA).

Robustness Analysis of Noisy Feature Attack

We further conduct experiments on ADNI datasets for the Alzheimer’s progression task by introducing Gaussian noise into the feature embeddings X extracted by SAM-Brain3D. The noise is defined as η = ρ ⋅ r ⋅ ϵ, where r is the mean of the maximum value of the feature embedding, ϵ ~ N(0, 1) is the Gaussian noise, and ρ is the noise ratio (set to 0.01 and 0.05). The experimental results are presented in Table 8. We observe that the accuracy and specificity remain stable under noisy feature attack. Though the F1 score and sensitivity can decrease, the performance is still better than or comparable to a simple discriminative classifier in #1 of Table 4 (F1 score: 62.27, sensitivity: 55.15). This demonstrates that our method maintains robustness to moderate noise.

Table 8.

Robustness analysis of noisy data for the Alzheimer’s progression task. We adopt the Gaussian noise attacks by injecting noise into feature embeddings, with ρ controlling the noise ratio.

Settings	ACC	F1	SPE	SEN
No Attack	88.09	70.23	96.43	62.12
Noise Injection (ρ= 0.01)	86.73	66.22	95.89	62.12
Noise Injection (ρ = 0.05)	86.37	65.56	95.87	54.20

Comparison with State-of-the-Art Methods

We compare our method with the following state-of-the-art methods, including both single- and multi-modal ones. The single-modal methods use PET modality, e.g., Multi-region Analysis [52], Multilevel Feature Representation [53], and 3D CNN [54]. The multi-modal ones include 1) Multimodal CNN [20] using concatenation for multi-modal fusion, 2) ProAuto-noAge [21] utilizing auxiliary MRI and PET data for multi-modal modeling, 3) Multi-scale graph [22] employing graphs for multi-scale fusion and a consistent predictions across different modalities, 4) HAN [23] using hypergraph for multi-modal fusion, 5) VAP-Former [24] adopting attention-based transformers to fuse both image and attribute modalities, and 6) the latest MMSDL [25] and GADL [55] leveraging multi-modal attention for heterogeneous data.

Table 9 shows the results of all the methods. From the results, we have the following observations. Firstly, our HyDA with the image encoder of SAM-Brain3D achieves the best accuracy and Specificity, beating the other competitors by 7.74 % in accuracy, and 4.45 % in Specificity. This large gap illustrates that our HyDA and SAM-Brain3D can outperform the other multi-modal fusion methods and obtain state-of-the-art performance for Alzheimer’s progression. Secondly, compared with single-modal method (Multilevel Feature Representation [53] and 3D CNN [54]), and multi-modal fusion using simple concatenation in Multimodal CNN [20] or auxiliary data in ProAuto-noAge [21], our hypergraph-based dynamic fusion surpasses them in accuracy, F1 score, AUC, and Specificity, usually by a large margin. This verifies the effectiveness of our HyDA in capturing high-order relations in multi-modal data. Our method is worse than them in Sensitivity, probably due to our highly imbalanced class (146 sMCI vs. 44 pMCI). Thirdly, the accuracy of our method is superior to Multi-scale graph [22] and the hypergraph method, HAN [23], while the F1 score is worse than HAN and the AUC is comparable to Multi-scale graph. It implies that our method can correctly classify the majority of cases, but may fall short in accurately recognizing the minority class. Finally, compared with the attention-based fusion, VAP-Former [24], MMSDL [25], and GADL [55], our method is clearly better in accuracy and F1 score. Particularly, our method manifests a significant improvement over the latest MMSDL in all five metrics. Overall, our method is highly competitive among state-of-the-art methods.

Table 9.

Comparison with state-of-the-art methods on ADNI. All the compared methods are based on deep learning.

Methods		ACC	F1	SPE	SEN	AUC
PET	Multi-region Analysis [52]	63.1	−	73.2	52.2	−
	Multilevel Feature Representation [53]	79.43	−	84.16	69.14	83.88
	3D CNN [54]	74.7	−	75.0	74.0	81.1
Multi-modal	Multimodal CNN [20]	72.22	−	71.25	73.44	77.49
	ProAuto-noAge [21]	78.70	−	80.00	77.30	77.56
	Multi-scale Graph [22]	80.60	−	−	−	85.50
	HAN [23]	71.10	76.66	−	−	76.19
	VAP-Former [24]	79.221	63.13	−	−	86.31
	MMSDL [25]	75.16	61.90	93.81	50.20	71.87
	GADL [55]	71.09	−	65.94	77.11	74.97
	SAM-Brain3D + HyDA (k = 20)	88.09	70.23	96.43	62.12	82.54
	SAM-Brain3D + HyDA (k = 28)	88.34	71.70	97.86	62.19	84.29

¹The class balanced accuracy (BACC) is adapted from the original paper rather than ACC.

5.3. HYDA for MGMT classification

We evaluate model performance using official metrics from the competition: Area Under the ROC Curve (AUC). Table 10 reports the results across five folds under different training strategies, including training from scratch (TFS), fine-tuning the rest of the network with the encoder frozen (FT-FL), and fine-tuning all parameters (FT-ALL).

Table 10.

Comparison of different training strategies (TFS: training from scratch, FT-FL: freezing encoder and training rest part, FT-ALL: fine-tuning all parameters) across models on 5-fold cross-validation. All results are reported as AUC scores (%). The last column reports p-values from bootstrap tests comparing each method to Our Method

Deep Learning Methods	Strategies	Fold AUC (%)					Mean ± Sth AUC	p-value (vs. Ours)
		Fold 1	Fold 2	Fold 3	Fold 4	Fold 5
EfficientNet	TFS	58.94	57.68	62.14	58.88	64.75	60.48±2.60	0.021*
ResNet	TFS	64.33	58.76	60.73	65.52	67.28	63.32±3.13	0.016*
	FT-FL	57.30	55.76	56.06	64.42	55.28	57.76±3.39	0.008**
	FT-ALL	56.70	52.58	49.30	60.55	60.56	55.94±4.44	0.049*
SegResNet	TFS	52.88	58.33	59.06	58.18	58.40	57.37±2.27	0.006**
	FT-FL	54.55	60.73	51.79	61.55	56.48	57.02±3.69	0.031*
	FT-ALL	56.39	62.67	59.61	60.30	58.83	59.56±2.04	0.065
Swin-ViT	TFS	59.14	57.50	53.94	59.30	60.19	58.01±2.21	0***
	FT-FL	47.15	58.36	53.67	58.91	54.78	54.57±4.22	0***
	FT-ALL	59.14	57.50	53.94	59.30	60.19	58.01±2.21	0***
SAM-Brain3D + GraphNet	FT-FL	65.27	61.26	57.92	56.30	59.88	60.13±3.08	0.002**
SAM-Brain3D + PopGraphNet	FT-FL	58.26	50.09	53.88	50.47	43.73	51.29±4.79	0.001**
SAM-Brain3D + HyDA-GCN	FT-FL	66.45	60.18	59.70	61.58	68.12	63.21±3.43	0.044*
SAM-Brain3D + HyDA-DHGNN	FT-FL	64.21	58.76	60.03	64.88	57.04	60.98±3.07	0.017*
Our Method	FT-FL	65.39	64.15	63.42	65.06	63.98	64.40±0.72	−

^*p<0.05,

^**p<0.01,

^***p<0.001.

Although our method does not achieve the highest AUC in every individual fold, it consistently ranks among the top in all folds and outperforms all baseline methods in terms of mean performance. Specifically, our method achieves the highest mean AUC (64.40) with the lowest standard deviations, demonstrating superior stability and generalization across folds.

Comparing the baselines, CNN-based models such as EfficientNet-B3-3D and ResNet50-3D trained from scratch show moderate performance but relatively high variance. Fine-tuning pretrained models (ResNet50-3D, SegResNet, and Swin-ViT3D) slightly improves individual results but does not consistently outperform training from scratch.

In contrast, methods integrating SAM-Brain3D as a feature extractor achieve notable improvements over traditional CNN and transformer-based models, including directly using a vanilla graph neural network as an adapter (SAM-Brain3D + GraphNet), replacing hypergraphs convolution networks with Graph Convolution Networks or Dynamic Hypergraph Neural Networks (denoted as SAM-Brain3D + HyDA-GCN and SAM-Brain3D + HyDA-DHGNN, respectively). This highlights the significance of leveraging high-quality, domain-specific pretrained feature extractors. Particularly, our method, based on SAM-Brain3D and HyDA, demonstrates the best overall generalization, achieving improvements in AUC while maintaining lower standard deviations.

The p-values further validate that SAM-Brain3D and HyDA enhance feature representation quality with statistical significance in most cases, leading to more discriminative and robust embeddings for MGMT promoter status classification.

6. Conclusion

In this paper, we introduce a brain foundation model called SAM-Brain3D to segment diverse brain targets from 3D medical images, including up to 35 brain structures. We further propose a novel Hypergraph Dynamic Adapter (HyDA) to adapt SAM-Brain3D to downstream brain diagnosis tasks, e.g., Alzheimer’s progression and MGMT classification. HyDA is designed to extract multi-modal information with hypergraphs and fuse multi-scale features with dynamic convolution, of which the kernels are dynamically generated and conditioned on each subject, facilitating subject-adaptive diagnosis. With SAM-Brain3D and HyDA, we achieve state-of-the-art performance on various segmentation and classification tasks for brain disease analysis.

However, our method may have the following limitations. Firstly, we rely on complete modalities for training and inference on downstream tasks, but acquiring all the modalities (e.g., MRI, PET, and non-imaging data) can be expensive. As such, many subjects may have incomplete modalities and these subjects cannot be used for training or inference, limiting the flexibility of HyDA. Secondly, our method cannot effectively handle the class imbalance issue. This leads to our method excelling in some metrics like accuracy and specificity while failing in the others (e.g., AUC or sensitivity) as shown in our experimental results. Thus, how to effectively utilize incomplete modalities and better tackle the class imbalance issue can be interesting for future work. Lastly, while our evaluation primarily focused on overall classification metrics, we acknowledge that model performance can vary across brain regions. Preliminary observations suggest that regions strongly implicated in disease pathology (e.g., the hippocampus in Alzheimer’s disease) tend to provide greater discriminative power, whereas regions with more subtle or heterogeneous changes are more challenging to model. A systematic region-level analysis, aligned with neuropathological evidence, could therefore provide valuable insights into both the strengths and limitations of our approach. We plan to pursue this direction in future work, with the aim of improving clinical interpretability and understanding of disease mechanisms.

Supplementary Material

Supplementary material associated with this article can be found in the online version at 10.1016/j.patcog.2025.112595

Data availability

The dataset that we use are introduced in the paper with the links and references.