The most important features in generalized additive models might be groups of features

Abstract

While analyzing the importance of features has become ubiquitous in interpretable machine learning, the joint signal from a group of related features is sometimes overlooked or inadvertently excluded. Neglecting the joint signal could bypass a critical insight: in many instances, the most significant predictors are not isolated features, but rather the combined effect of groups of features. This can be especially problematic for datasets that contain natural groupings of features, including multimodal datasets. This paper introduces a novel approach to determine the importance of a group of features for Generalized Additive Models (GAMs) that is efficient, requires no model retraining, allows defining groups posthoc, permits overlapping groups, and remains meaningful in high-dimensional settings. We showcase properties of our method on three synthetic experiments that illustrate the behavior of group importance across various data regimes. We then demonstrate the importance of groups of features in identifying depressive symptoms from a multimodal neuroscience dataset, and study the importance of social determinants of health after total hip arthroplasty. These two case studies reveal that analyzing group importance offers a more accurate, holistic view of medical issues compared to a single-feature analysis.

Introduction

As machine learning models become ever-more ubiquitous in healthcare, there is a growing necessity for model interpretability. While it has proven difficult to reach a consensus on the meaning of interpretability¹, one particularly common instrument is determining the importance of individual features (aka feature importance or feature attribution), which in and of itself also does not have one single definition. Common definitions of feature importance include the average contribution of a feature to prediction in a separable model^2–4, average impurity decrease in tree-based models^5,6, and the average increase in the model’s error or loss after shuffling the values of a feature (i.e., permutation feature importance)^7,8.

Providing an interpretation based on the importance of individual features is often challenging due to feature correlation and plausibility issues⁹. For a group of correlated features, their joint signal could be crucial for prediction, while the individual features in the group might be attributed low feature importance¹⁰. As a result, there could be signal in groups of correlated features that might be lost when analyzing individual features by themselves. Even worse, these features could be filtered out during feature selection and thereby diminish the signal for inference⁶. This could be particularly troublesome on data containing natural groupings of features, such as modalities in multimodal data. Furthermore, interactions between features (in a group) are not captured through feature importance¹¹. One way to address these issues is via group importance (aka grouped feature importance), i.e., by grouping features together and determining the importance of the group^12,13.

Group importance has previously been proposed using LASSO¹⁴, the Penalty Decomposition¹⁵, Shapley values¹⁶, and permutation testing^8,17. Unfortunately, most of these methods come with computational challenges, mainly originating from the need to retrain the underlying machine learning model or compute expensive gradients. In the case of groupSHAP¹⁶, for example, group-level attributions require estimating Shapley values over subsets of features or feature groups, which involves evaluating the model on a large number of masked feature combinations. Exact Shapley values require an exponential number of masked combinations in terms of the number of features. While efficient approximations of Shapley values exist for tree-based models, e.g. TreeSHAP, Shapley values remain computationally expensive for general models and large datasets.

To address these challenges, we propose a robust, versatile, and computationally efficient method for computing group importance for Generalized Additive Models (GAMs), a very broad class of interpretable models that includes linear, logistic, and generalized linear models. GAMs with interactions^2,3 are typically of the form , where g is a link function, y the target, are features, is an intercept, and and are shape functions (aka component functions). In this setting, the (individual) importance of a feature is typically computed as the average absolute contribution of the corresponding shape function ⁴, i.e.,

1

where represents samples t in training data . For a group consisting of both individual features as well as interaction terms , we extend this definition and propose

2

with mathematical justification given in Section "Group importance for GAMs". This definition of group importance has several properties we consider valuable, including:

• groups can be defined post hoc (and thus do not affect the model) and may be overlapping,

• computing group importance is computationally inexpensive due to the separability of the GAM,

• it remains meaningful when there are many correlated features, unlike, e.g., group permutations,

• for a group consisting of one feature, its group importance equals the feature’s individual importance,

• adding useless features, i.e. random noise, to a group does not change its group importance (up to small error ),

• both individual features and interaction terms can be included in the group G.

We implement our method for an Explainable Boosting Machine (EBM), a GAM-based model that achieves state-of-the-art accuracy while providing interpretability^2,4. EBMs have recently gained popularity for their application in fields such as concrete compressive strength¹⁸, brain tumors¹⁹, Parkinson’s disease²⁰, and revealing healthcare biases that result in significant loss of life²¹. We evaluate the properties of our definition of group importance on three synthetic experiments and two real-world datasets that contain natural groupings of features. In the first case study, we identify the depressive symptom of negative valence^8,22 on a neuroscience dataset consisting of measurements from functional brain networks and groups (aka domains) such as sleep, traumatic experiences, biographic data, and social support. In the second case study, we focus on the influence of community-level social determinants of health in predicting mortality for total hip arthroplasty using a large-scale healthcare dataset (k samples). These determinants, encompassing environmental, economic, and social conditions, are paramount in understanding health outcomes. Our main contributions are as follows:

• Proposing a definition for group importance for Generalized Additive Models that is computationally inexpensive.

• Showing that group importance offers enhanced interpretability in addition to individual feature importance, especially for correlated features, as demonstrated on two real-world datasets.

• Demonstrating group importance can be used as a low-cost, yet effective method that achieves similar importances as groupSHAP in a real-world dataset¹⁶. It is also a low-cost method for feature selection.

Methods

We first describe Explainable Boosting Machines (EBMs) as a specific implementation of a Generalized Additive Model (GAM), and how feature importance is typically computed in this setting. Then, we extend this approach to define group importance. Subsequently, we turn to feature selection and briefly discuss the parallel between group importance and explained variation in statistics.

Explainable boosting machine

Explainable Boosting Machines (EBMs)² are Generalized Additive Models (GAMs)³ with interaction terms, i.e.,

3

The shape functions and are trained using cyclic gradient boosting with shallow learners². During training, a purification process^4,23 ensures that the contribution of a feature for prediction is not moved between and its interaction terms for . That is, encodes the sole and total contribution of to prediction (i.e., the main effect), while encodes the interaction term adjusted for all main effects.

Thus, the shape functions form a functional ANOVA decomposition^23,24 through a functional ANOVA projection so that all lower-order effects integrate to zero, ensuring each main effect and interaction captures only its unique contribution. For enhanced interpretability, these functions are zero-centered and can be visualized to show the contribution of each feature or interaction term to prediction. Assuming that the GAM satisfies the functional ANOVA decomposition (e.g., after purification), there is no projection or correlation between higher-order interaction terms and lower-order interactions or univariate terms. For example, 3-bit parity is uncorrelated with and has no projection on 2-bit parity or the constituent univariate features. Thus if the 3-bit parity interaction is not included in the model, the contribution of 3-bit parity to the true generating function is left unmodeled and does not “pollute” the lower-order terms. This is part of what makes functional ANOVA useful for the interpretation of GAMs and why the EBM algorithm is designed to approximate functional ANOVA. It is also why purification is important when the approximation to the functional ANOVA is poor^4,24. In the absence of purification (and functional ANOVA), the signal of important higher-order interactions that are not included in the additive structure of the GAM may be distributed across lower-order terms of the corresponding features. This potentially affects the interpretation of both individual and group-level importances, which is a limitation if the functional ANOVA (e.g., purification) is not employed^2,4,24.

EBMs have been shown to be stable even under a low prevalence regime of the target, e.g. in credit card fraud (0.17%)²⁵, various adverse pregnancies outcomes (0.4% – 3.4%)²⁶, and intensive care unit readmission (8.5%)²⁷. However, if the sample size is so low relative to the number of features and interaction terms included in the model that the shape functions themselves become unreliable, then the reliability of the feature importances also will be reduced.

The additivity of the GAM makes it easy to define an importance metric. That is, individual, ‘global’ feature importance is typically defined as the average absolute contribution of a feature to prediction over all samples in the training dataset⁴. Notably, feature importance can sometimes refer to local feature importance, which attributes importance to features for one sample prediction specifically⁴, but we only consider global feature importance in this paper. That is, for feature and samples taking on values , the feature importance ⁴ of is typically defined as

4

Note, each contribution can include weights defined by the user⁴. This approach not only provides insights into the internal mechanisms of the model, but also ensures a symmetric attribution of importance, i.e., features signaling very high or low risk receive similar importance (a direct consequence of the zero-centering in the shape functions). Notably, the importance of a feature is not the cumulative contribution of all model terms that contain ; instead, its feature importance is defined solely as the average absolute contribution to prediction of the main-effect shape function , which is adjusted for all other model terms. Feature importance for GAMs under the functional ANOVA assumption is different from feature importance for alternate methods such as SHAP.

Correlated features

Note that, by design, EBMs split the shared signal of highly correlated features evenly. This ensures that each feature receives a proportionate amount of signal and importance. This is a property worth highlighting, as there also exist alternate approaches. For example, other GAM implementations might explicitly aim to assign all importance to one of the features rather than splitting it evenly, such as in GRAND-SLAMIN’²⁸.

Group importance

Similar to individual feature importance, group importance has been well-studied and has multiple definitions, including through Shapley values¹⁶ and permutation testing^8,17. Group importance is understood differently across various domains, each with its own set of definitions and accompanying challenges^12,13. This lack of consensus might lead to arbitrary selection of definitions rather than standardization. We aim to bridge this gap for GAMs specifically and propose a definition that naturally extends the notion of individual feature importance.

One example of a definition of total importance could be a simple sum of individual importances, but this leads to a misleading scale when features exhibit substantial correlation, see Section “Synthetic experiments”. Another common definition is groupSHAP¹⁶. While (group)SHAP is very popular, such group-level attributions require estimating Shapley values over exponentially many subsets of features, which involves evaluating the model on a large number of masked feature combinations, resulting in potentially severe computational challenges.

Group importance for GAMs

To post hoc evaluate the importance of a group of features after training the GAM, we average the absolute values of the sum of feature contributions in a group over all samples in the training dataset . That is, let group

be a set of features whose group importance we wish to evaluate. Then, we define group importance as

5

This is a canonical choice given that

1. it is a direct extension of the case, with the property that a feature’s individual importance equals the importance of a group consisting of only that feature,

2. the null group (i.e. group consisting of no features) has 0 importance,

3. group importance satisfies .

For (ii), the null group has zero importance as the summand in Eq. (5) is always zero. Observe that (iii) is a particularly important and useful property, as it represents the triangle inequality for grouped feature importance. As such, (iii) naturally extends to groups: if are groups of features and we let , then . This is a direct result of the definition in Eq. (5). Note, another natural extension of group importance would have been as the summand. However, this would result in group importance simply being the sum of feature importances, as

A benefit of this alternative definition is that it is perfectly additive for non-overlapping groups. In our proposed definition, group importance scores are not additive: summing group importances as we propose them corresponds to summing up absolute values (averaged over training samples), whereas the proposed definition of group importance is defined as the absolute value of a summed contribution (also averaged over training samples).

Similar to other definitions of group importance^8,14–17, our definition is permutation invariant. That is, for any permutation , the symmetry group, it immediately follows from the definition that . Besides permutation invariance, our notion of group importance is also invariant under linear feature scaling: linearly scaling the values of a feature correspondingly stretches the shape function, thus leaving feature and group importance invariant. However, under nonlinear transformations (e.g. the log transform), feature importances may be affected.

In the same vein, having a different number of features between groups does not artificially inflate the larger group’s importance, since all shape functions are zero-centered and we adopt the definition in Eq. (5) rather than the (flawed) alternative definition above.

Including higher order interaction terms

Since the interaction terms learned by a GAM are adjusted for the individual effects (assuming purification²³), group importance can be computed over any set of individual features and interaction terms. This holds true for both second order interactions, and for higher order terms if they were included in the GAM, which is not traditionally the case^2,3. For any GAM with higher-order interactions,

6

we can define any group G of features and interactions to compute the feature importance over. For example, with indices , where each group of features corresponds to the shape function in the GAM. Notably, if , this term reduces to the individual shape function . For any group G and its feature indices , our notion of group importance naturally extends to

7

In the example of and , the group importance reduces to

8

As a concrete example, consider individual features , their feature interaction , and a target . The main effect of , adjusted for all other individual features and feature interactions, contains no information about the XOR target. Consequently, is flat and the importance of is zero, i.e. . The same holds true for : after adjusting for other terms, it carries no predictive signal for the XOR target, yielding a flat shape function and . Since both main effects are identically zero, the group consisting of the two individual features has importance

as the two flat shape functions contribute no signal jointly. In contrast, the feature interaction can perfectly encode the target, because jointly observing and fully determines . Its corresponding shape function is therefore non-flat and yields . The power set of the features has group importance . Thus, , while . These importance scores align with the ANOVA interpretation that XOR is captured solely by the interaction term.

Because the mechanics of including an individual feature in a group importance computation are the same as including a higher-order interaction term, the remainder of the manuscript continues to focus without loss of generality on groups of individual features.

Explained variation and feature importance

We contrast feature importances with an alternate measure of importance in statistics: explained variation (aka explained variance). Explained variation represents to what extent a set of variables in a model accounts for the variation (often variance) of a target. In regression problems, explained variance is typically defined as , the coefficient of determination²⁹. For classification problems, however, it depends on the setting, but is usually also a measure of ‘goodness of fit’³⁰.

In Principal Component Analysis (PCA)³¹, the scale for explained variance is the sum of the individual component variances . For independent variables (e.g., principal components) , the variance explained by is then defined as ³². More generally, if the are independent or at least weakly correlated, can be approximated by the total variance . While one might loosely view total variance as analogous to the group importance on the full feature set, there exist pathological settings in which there exists a group G that is a strict subset of all features such that . Even the maximum group importance attainable over all feature subsets is not analogous to total variation, as this would imply comparing a group of features to a potentially different set of features, which is nonsensical. The sum of individual importances (plus importance of interactions) is also not appropriate as this scale is generally not achieved unless all feature importance contributions are perfectly additive.

Feature selection using group importances

Once relative group importances have been computed, or a ranking has been obtained more generally, this can also be used for feature selection. That is, by picking the top k groups of features that are assigned most importance by the model. This is particularly useful when feature selection has to be performed at a group-level for predefined groups. This is important because in many practical applications individual features cannot be selected independently.

For example, in the case of a budget-constrained finance company choosing which datasets to acquire from a data vendor, they often get to trial and evaluate each dataset before choosing which to acquire. Once you determine which ones to buy, you can only select the features available in said datasets and not in the others. Thus, features are selected only at a group level, and it is not possible to select the top individual features from each dataset. Other examples where features can only be selected in predefined groups include medical lab values, where each lab test results in multiple lab values, or choosing what questionnaires to give patients during a psychology study.

Synthetic experiments

To demonstrate the behavioral properties of our proposed definition of group importance for GAMs, we generate three synthetic datasets. We examine the behavior exhibited when features contain signal that is (1) additive and perfectly correlated, (2) opposing and independent, and (3) opposing and highly correlated. One real-world healthcare example of opposing (and correlated) signal can be found in age-related diseases: advanced age indicates increased risk, while a sufficient amount of medication lowers risk, therefore containing opposite signal. At the same time, age and the amount of medication are likely correlated.

Generalizing from these three extreme, synthetic cases, we then study group importance as a function of correlation in both the additive and opposing signal case. We also review various edge cases in Supplementary Information S1. Note, the EBMs achieve near-perfect prediction accuracy in each synthetic experiment due to the simplistic setup of the synthetic data. The synthetic datasets consist of samples , where is sampled from the uniform distribution; we define and in each subsection. We denote and .

Additive signals

To create synthetic data consisting of additive signals, let z be a copy of x. Furthermore, let be noise sampled from the normal distribution in order to construct label if and 0 else. No matter whether or , z and x are equally indicative of any target y.

Then, due to perfect correlation, and each learn an identical signal that is half of the total signal in the data (see Fig. 1(a)), as described in Section “Correlated features”. Thus, the group importance of x and z is equal to the sum of their individual importances, i.e., (see Fig. 1(a)). Similarly, if a feature x is duplicated and negated to , the corresponding shape functions and contain negated, yet equivalent, signal and therefore also have equal feature importance. This is a direct consequence of the scaling property discussed previously. This example is shown in Supplementary Information S1.1.

Fig. 1.

Group importance of the Explainable Boosting Machine (EBM) on various synthetic datasets with two variables x and z, and target y. By design, these variables can either be additive or conflicting in predicting target y; that is, the shape functions of x and z could have similar or opposing shapes. Part (a) shows additive signal through z=x. Part (b) shows independent, conflicting signals, while part (c) shows correlated, conflicting signals. While the group importance ranking in (b) is larger than each single feature importance, the group importance in (c) shows that the two signals almost always cancel each other since the features x and z are highly correlated and close in value. Part (d) demonstrates group importance as a function of correlation for additive signals, while part (e) visualizes group importance as a function of correlation for conflicting signals. Note that the case of zero correlation in part (d) corresponds to the experiment in part (b). High correlation in part (e) corresponds to the experiment in part (c), and the extreme case of perfect correlation in part (e) is explored in Supplementary Information Fig. S3.

Independent, conflicting signals

To obtain a feature z that is independent of x, sample . Then, to create opposing signal of z with respect to x, let if and 0 else. We visualize the shape functions and and show the individual and group importances in Fig. 1(b).

By symmetry in the construction of , an increase in increases the probability that (for random ), while an increase in decreases that probability (for random ). As a result, the shape functions and in the EBM are each other’s additive inverses, and their group importance is less than the sum of individual importances.

To analyze the behavior shown in the group and feature importances in a more detailed fashion, recall that the group importance of features x and z is formulated as

9

In this synthetic experiment, features x and z contain exactly opposite signal, i.e., . Because feature values are independently sampled, there might be many samples t where and (and vice versa) such that these terms do not cancel each other out and, as a result, contribute to the group importance. Similarly, there are roughly equally many samples such that and , as well as and , assuming large n. Thus, the group importance is slightly larger than and , but far smaller than their sum .

Correlated, conflicting signals

We now examine how our proposed definition of group importance behaves for correlated features that contain opposing signal. This is not an uncommon scenario in practice as, in many instances, features tend to be correlated and samples be drawn non-iid.

To have z be correlated with x, we let , where we sample (and scale to [0, 10]), so that x and z are highly correlated. To retain conflicting signal, we again define if and 0 else.

The shape functions remain mostly unchanged with respect to the previous synthetic experiment, and the individual feature importances also remain balanced. However, in Fig. 1(c) we see that the group importance . This is a direct result of the correlation and, in particular, for all samples and , meaning the shape functions nearly cancel each other out for most samples. That is, for perfectly opposing signal: the more correlation, the lower the group importance compared to the individual feature importances. In general, if feature is such that , then the features must contain opposing signal as a whole compared to , since they lower the mean absolute contribution to the target.

Group importance as function of correlation

In the three synthetic examples above, we considered the extreme cases of perfectly additive or conflicting signal with either high or no correlation between feature values. We now generalize these experimental setups by computing group importance as a function of the Pearson correlation between x and z. This is achieved by varying boundary such that yields a nearly uniformly distributed on one end, or a z perfectly correlated with x on the other end of the spectrum. (We again scale to [0, 10] for convenience in constructing y.) In other words, we vary the correlation between x and z and study the effects on their group importance, in both the additive and conflicting setup.

Additive signal

To generalize the experiment on additive signals in Section “Additive signals”, we keep the same target y and compute the group importance as a function of the Pearson correlation between x and z (see Fig. 1(d)). We also show the individual importances of x and z given the asymmetry in how y is constructed. Observe that the total group importance is constant (up to random noise) since target y is determined using only feature x and not z (see Fig. 1(d)). For perfect correlation (i.e., as in Section “Additive signals”), we see that x and z are both assigned half the total, group importance, while y has near-zero importance when it is near-uniformly distributed and uncorrelated with x.

Conflicting signal

Generalizing the conflicting signals experiment of Section "Independent, conflicting signals", we visualize the group importance as a function of the correlation in Fig. 1(e). We observe a roughly linear decline in the group importance of x and z as their correlation increases, transitioning to a more exponential decrease at higher correlation levels. This pattern likely arises from measuring group importance as the mean absolute contribution, which inherently slows the convergence to zero. However, the group importance does converge to zero as the correlation goes to one.

Real-world experiments

We evaluate the computational efficiency of group importance and demonstrate the utility and relevance of group importance in two real-world medical datasets. In our first case study, we identify the depressive symptom negative valence based on groups of correlated features drawn from functional brain networks and various life and behavioral domains. As a second case study, we investigate the importance of social determinants of health in the outcome of hip replacements. For each we analyze the group importance ranking, and use this as a guide for applying group importance for low-cost feature selection purposes.

Computational efficiency

Unlike other group importance methods, our method is computationally low-cost. It requires only k function evaluations for each sample , resulting in a complexity of , or in terms of function evaluations. This is an order of magnitude faster than the popular grouped permutation importance (GPI) method¹² and groupSHAP¹⁶ across different experimental setups, even if the number of features or samples is very high. For various OpenML datasets with such different data regimes³³, there is an order of magnitude difference in terms of runtime (on an Intel(R) Xeon(R) CPU @ 2.20GHz with 12GB RAM) of group importances implemented for GAMs versus GPI and groupSHAP (see Fig. 2).

Fig. 2.

Computational cost in terms of runtime of various metrics of group importance. Our method is compared to Grouped Permutation Importance (GPI)¹² and groupSHAP¹⁶ for a variety of datasets picked to represent different characteristics, such as number of data samples and variables. These seven datasets, annotated with the number of rows and columns in each dataset, were downloaded from OpenML, an open-source platform for sharing datasets that is commonly used for benchmarking machine learning models³³. We were unable to reproduce the runtimes for the permutations-based ranking⁸, and the runtime of groupSHAP for Higgs and GCM was cutoff at . Our method is an order of magnitude faster to compute on all OpenML datasets but GCM, where it is very marginally better. The experiments were performed on an Intel(R) Xeon(R) CPU @ 2.20GHz with 12GB RAM.

Neuroscience experiments

While better understanding the underlying mechanism of depressive symptoms could hugely improve mental health care, this is a difficult task as the manifestation of the symptoms is quite heterogeneous. That is, they are not confined to a single, easily identifiable brain region or network, nor clear changes in behavior^34–36.

Data

We conduct our analysis on publicly available data acquired by the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA)³⁷ (Baseline to Year 6 Data Release of Collection 4513 from the NIAAA Data Archive, https://nda.nih.gov/edit_collection.html?id=4513.

). The NCANDA dataset contains multimodal and longitudinal data comprised of biographical, self-reported behavioral, and fMRI data of 1396 observations (aka visits) from 521 participants (ages 12 to 18 years).

Predictors There are 23 resting state fMRI (rs-fMRI) features as well as 131 demographic & behavioral measurements that are categorized into eight disjoint groups^8,37:

1. Life & Trauma Events (full: Life events and childhood trauma): encompasses trauma, emotional neglect, and (parental) marital separation

2. Personality Traits: include agreeableness, emotional stability, extraversion, positive thinking, and cognitive restructuring.

3. Neuropsychological Battery: includes problem-solving and reasoning skills, attention, memory, and language³⁸.

4. Executive Function Spectrum (full: Behavior Rating Inventory of Executive Function (BRIEF)): assesses executive functions inhibition and shifting, measuring the ability to resist or delay impulses and to tolerate change³⁹.

5. Social Support Features: records relationships with friends and family, as well as active participation in social clubs and events.

6. Sleep Patterns: tracks wake-up time and bedtime for weekdays and weekends, as well as sleepiness and circadian preferences.

7. Demographics (full: Demographics and pubertal development): includes age, sex, BMI, ethnicity, pubertal development scale score.

8. Alcohol & Drug Use Features: records alcohol and substance use.

The 23 brain functional measurements were generated by processing each resting-state fMRI (acquired on a GE or Siemens scanner; voxels: 64x64x32, voxel size: 3.75mm x 3.75mm x 5 mm; TR: 2.2s) using the NCANDA pipeline outlined in⁴⁰. The pipeline consists of skull-stripping, motion correction, detrending, spatial smoothing, and anomaly detection through a manual quality check. All scans of this analysis had a minimum of 7.5 minutes of usable scan time and were registered to the SRI24 atlas⁴¹. 23 intrinsic function networks (IFN) were extracted via longitudinal independent component analysis (ICA), and each IFN is a brain network that regulates different aspects of our cognition and behavior, such as memory, attention, emotions, and motor skills. Each of the 23 IFNs is assigned a score representing its strength, i.e. the average functional connectivity within the brain network, sometimes referred to as the functional network’s efficiency score⁴². To account for scanner differences in the acquisition (GE or Siemens), we regress out scanner type from the efficiency scores by performing scanner-level standardization.

The eight disjoint, non-fMRI groups defined above, and its features, are significantly correlated^8,37,43. This may affect the individual importance scores, as such features share correlated signal. However, the correlation does not hinder the interpretation of the group importances; in fact, our proposed group importance is able to account for correlation (see Section “Synthetic experiments”).

Target variable We identify negative valence^8,22, an aggregate statistic for feeling fear, anxiety, sadness, loss, and threat, as a measure of depressive symptoms. The negative valence target has been extracted from the National Institute of Mental Health (NIMH) Research Domain Criteria (RDoC) initiative, which is a research framework for approaches to investigating mental disorders^44,45. It has been shown that negative valence’s subdomains are highly correlated with Major Depressive Disorder (MDD) and that MDD reflects dysfunctions in negative valence systems⁴⁶. Negative valence has a prevalence of 6.02% across all visits.

Model (EBM) configuration The EBM was trained using 100 outer bags, 50 inner bags, 25 min samples leaf, 64 max bins, and 500 smoothing rounds (to prevent overfitting).

Group importance ranking

To assess the accuracy achieved by the EBM, we conducted stratified 10-fold cross validation. All visits by a subject were consistently in either the training or test set in order to avoid information leakage to the test set. This is necessary since the data is longitudinal in nature. The EBM model attains a mean test AUC of , balanced accuracy of , and Brier score of across visits, where the errors show the standard error of the mean (SEM). This is on par with models such as random forests, recurrent neural networks, and logistic regression trained on this dataset⁸. The strong predictive performance of the EBM model shows that the learned shape functions are discriminative. The results in Table 1 show the stability of the derived statistics, such as feature importances. As with most machine learning models, an EBM is not intended to recover causal relationships or the true data-generating process; rather, its value lies in providing transparent predictions from the available data and features.

Table 1.

Ranking of the group importances of the groups of features in the neuroscience case study using NCANDA data³⁷. The 23 resting state fMRI (rs-fMRI) features and 131 demographic and behavioral measurements were grouped into nine disjoint groups^8,37. The permutations-based ranking has been previously reported⁸ on NCANDA data excluding rs-fMRI brain network scores. The group permutation importance (GPI)¹² and groupSHAP¹⁶ metrics were also used for comparison. While there is a lot of similarity in the various rankings, our ranking is consistent with the literature^47–51, and is most similar to the ranking obtained by groupSHAP, with the top three most important features being identical. All methods agree on the two most important groups, i.e., Life & Trauma Events and Personality Traits, while Demographics and Alcohol & Drug Use are the least important across all metrics. To evaluate the statistical significance of the ordering of features, a Friedman test was applied on the full group rankings as well as on the top three features per method, showing statistical significance. On the other hand, the Wilcoxon signed-rank test further indicates that the relative ordering of the top two groups is not statistically distinguishable for any of the methods considered.

Rank	Ours	groupSHAP16	GPI12	Permutations8
1	Life & Trauma	Life & Trauma	Personality	Personality
2	Personality	Personality	Life & Trauma	Life & Trauma
3	Neuropsychology	Neuropsychology	Executive Function	Executive Function
4	Executive Function	Brain Networks	Social Support	Sleep Patterns
5	Social Support	Executive Function	Neuropsychology	Alcohol & Drug Use
6	Sleep Patterns	Sleep Patterns	Sleep Patterns	Social Support
7	Brain Networks	Social Support	Brain Networks	Neuropsychology
8	Demographics	Demographics	Demographics	Demographics
9	Alcohol & Drug Use	Alcohol & Drug Use	Alcohol & Drug Use
p-value top 2	0.641	0.275	0.439
p-value top 3	0.031	0.023	0.045
p-value full ranking

The most important individual feature was emotional stability, a feature belonging to the Personality Traits group. Since there are significant correlations between features and groups in NCANDA^8,37,43, we aim to gain a deeper insight into which groups of features were driving prediction. To that end, we combined all fMRI features into one group and divided the rest of the measurements into the eight groups described in Section “Data”. This yields the group importances shown in Fig. 5, revealing that five of the nine groups are more important than the most important individual feature.

Interestingly, the most important group is Life & Trauma Events while only five of its features are among the 20 most important features. The group Life & Trauma Events contains mostly highly correlated features, namely chronic/non-chronic life aspects that are controllable/uncontrollable and have a positive/negative/ambiguous outcome (each of the combinations is a feature). Naturally, these features are highly correlated and are thus each attributed relatively low feature importance. Yet, when considered collectively, these features form the most critical group for prediction, highlighting the significance of group importances in our analysis. This phenomenon of high group importance for correlated, low-importance features was also observed in the additive synthetic experiment (Section “Additive signals”) and its generalization (Section “Additive signal”).

Table 1 presents the rankings of the NCANDA groups based on their group importance, as determined by several methods. Our method’s ranking is compared to those obtained through groupSHAP¹⁶, Grouped Permutation Importance (GPI)¹² and a previously reported ranking on the NCANDA dataset, derived using another permutation-based method⁸. While there is significant overlap in rankings, our method appears to attribute much more importance to neuropsychology battery than the permutations-based ranking⁸, ranking it as the third most important group instead of the seventh. GPI ranks neuropsychology as the fifth most important group. Our higher ranking seems more in line with the literature, which extensively links depressive symptoms to measurements of the neuropsychological battery group, including reasoning skills^47,48, memory⁴⁹, and intellectual functioning^50,51. Notably, our ranking is most similar to that of groupSHAP. All four methods agree on the two most important groups, namely Life & Trauma Events and Personality Traits, while Executive Function is also found to be an important group. On the other end of the spectrum, the group Demographics is attributed low group importance across the board, with its individual features also being attributed low individual feature importance. Similarly, our method, groupSHAP, and GPI rank the group Alcohol & Drug Use as the least important group, while it is the 5th most important group according to the permutations-based ranking⁸.

One group omitted by Paschali et al. (2022)⁸ was Brain Networks. Given that functional interactivity between brain networks tend to be correlated⁵², just like in the case of Life & Trauma Events, this decision might have been based on analyzing low feature importances. However, our analysis reveals that the importance of the group of fMRI features is substantial and is larger than all individual features but emotional stability, while also outranking the two groups Demographics and Alcohol & Drug Use.

To evaluate the significance of the rankings, we evaluated ranking separability and stability through statistical tests across 10 folds. This is done for our proposed group importance metric, groupSHAP and GPI; we were unfortunately unable to reproduce the Permutations baseline⁸. Using a Friedman test on the full group rankings, we obtain statistically significant results () for the proposed method, the GPI baseline, and groupSHAP¹⁶, indicating consistent rank separation among feature groups. Focusing on the top three groups, the Friedman test yields for all three methods. The Wilcoxon signed-rank test further indicates that the relative ordering of the top two groups is not statistically distinguishable for any of the methods considered. The exact p-values can be found in Table 1.

Feature selection

Following the analysis of the group importance ranking, our next step is to evaluate the practicality of using this ranking for feature selection. We conduct a series of experiments where we first train the EBM using the top groups, assessing model accuracy with test AUC and the Brier score through 5-fold cross-validation. The cumulative importance of these groups as they are sequentially added in order of their importance can be found in Fig. 3.

Fig. 3.

Cumulative group importance when successively adding feature groups. We compare the proposed group-level ranking to both the sorted order of individual feature importances and a greedy forward selection strategy. As expected, group-level selection exhibits a stepwise pattern and is less efficient on a per-feature basis than greedy forward selection, which operates at the level of individual features. The purpose of this comparison is to illustrate the behavior of group-level selection when features can only be selected in predefined groups, showing that selecting meaningful feature groups can provide a practical approximation to forward selection under such constraints.

Accuracy for top groups: Based on the ranking of groups as defined in Table 1 by our method, we compute the average test AUC and Brier score when trained on the top groups respectively (see Fig. 4). Training with just the top three groups achieves a test AUC and Brier score almost comparable to using all nine groups. Figure 4 shows that additional groups of features did not appear to significantly increase predictive power once (at least) the first three groups of features are used for training. However, Fig. 5 shows that the remaining six groups are assigned non-trivial group importance each. This implies these groups make non-negligible contributions to the model output through the learned shape functions, even though they are, as a whole, significantly correlated with the groups of Life & Trauma Events, Personality Traits, and Neuropsychological battery. Figure 4 shows there is barely an improvement in the AUC and Brier score after including these three groups.

Fig. 4.

Mean area under the ROC curve (AUC) and Brier loss, which measures model calibration, when the EBM is trained on the top 1 through 9 groups of features ranked by group importance. Using just the top 3 groups yields nearly optimal performance, indicating feature selection may be performed effectively on a group level.

Fig. 5.

Group importance scores computed using our method for identifying depressive symptoms using data from the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA)³⁷. The groups Life & Trauma Events and Personality Traits are the most important according to model attribution, highlighting the significance of behavioral and environmental factors.

Healthcare experiments

In our healthcare case study, we evaluate the importance of community-level social determinants of health variables in predicting mortality after total hip arthroplasty. It is debated in the literature how important patient demographics, such as race, sex, and age are compared to broader, community-level social determinants of health (SDOH), like healthcare access and socioeconomic status^53–55. In this section we compare an aggregate measure of SDOH, i.e. ‘community’, to various individual factors related to demographics as well as the Elixhauser Comorbidity Index.

Data

Our healthcare study utilizes data extracted from the Pennsylvania Health Care Cost Containment Council (PHC4) Database, covering the period from 2012 to 2018. This comprehensive dataset encompasses de-identified patient information, diagnostic and procedural codes, and financial data from 170 non-governmental acute care hospitals in Pennsylvania. Specifically, we focused on 140,092 patients who underwent elective primary Total Hip Arthroplasty (THA) according to validated International Classification of Diseases codes. After applying exclusion criteria such as inflammatory arthritis, non-elective admissions, and demographic omissions, our final cohort consisted of 105,336 patients. Key patient- and facility-level variables were extracted, including demographics and medical comorbidities indexed by the Quan adaptation of the Elixhauser Comorbidity Index. Unfortunately, certain granular, patient-level information such as a patient’s income was unavailable, although it has been shown that socioeconomic status is very correlated with income and poses a reasonable proxy⁵⁶.

Community-level variables were an integral part of our analysis. These were extracted by geocoding the residential 5-digit zip codes of each patient to their corresponding census tract variables from the American Community Survey (ACS). We focused on variables known to influence THA outcomes, including social support indicators (percent living alone), immigration and acculturation factors (percent foreign-born, percent non-English speakers), digital literacy (percent with computer and internet access), socioeconomic status (median household income, percent uninsured), and education levels. Additionally, we included the National Walkability Index, given its relevance to osteoarthritis and post-arthroplasty outcomes. These variables collectively formed the ‘community factors’ group in our analysis, allowing us to assess their aggregated impact alongside individual patient factors such as race.

Target variable: Our target variable is 90-day mortality, a critical measure in hospital settings due to its direct relevance to patient outcomes and healthcare quality. Mortality occurred in 0.3% of cases.

Model (EBM) configuration: The EBM was trained using default parameters.

Results

Our EBM obtains an average test AUC of 0.76. Similar to the neuroscience case study, we find that it is in fact a group of individually weak features that is driving the prediction of 90-day mortality, not necessarily individual features. We compute the feature group importance of the group of community-based features, and contrast this in Fig. 6 with the individual feature importance of race, sex, age, discharge location, and the Elixhauser Comorbidity Index.

Fig. 6.

Group and individual feature importances for predicting 90-day mortality after total hip arthroplasty on the Pennsylvania Health Care Cost Containment Council (PHC4) database. Community-level social determinants of health collectively are attributed larger importance by the model than individual predictors such as age, sex, and comorbidities, traditionally considered the main risk factors^53,57–61. Note that feature and group importances reflect the importance of these terms in the learned model, not necessarily the causal importance of these factors in the true generating function.

Remarkably, the most important feature for predicting mortality in hip arthroplasty is the group of community-based measurements, carrying x the importance of Race, a well-recognized risk factor^53,57,58. Moreover, the community-level SDOH features are assigned more importance than sex and age combined, while these are also considered important risk factors for adverse outcomes after undergoing total hip arthroplasty^59–61. The second most important feature is discharge location, which is an important predictor that could be informative of the amount of care available to a patient. It has been shown in the literature that patients discharged to a nursing or rehabilitation facility are less likely to have severe adverse events (e.g., mortality) compared to those who attempt to self-manage at home⁶².

These findings underscore the importance of considering social determinants of health in hip arthroplasty outcomes, and beyond in the healthcare system. Despite the tendency for social determinants to be highly correlated^63,64, which often leads to low individual feature importance, their collective influence as a group is substantial. This reinforces the value of group importance analysis in revealing critical insights that might be obscured when focusing solely on individual feature importance.

Ethical considerations

The inclusion of sensitive features such as race, trauma-related measures, or community-level socioeconomic indicators warrants careful ethical consideration. While such features can be controversial, excluding them does not eliminate their influence on model predictions, as correlated features may still encode similar information implicitly. In this sense, transparent models such as EBMs allow the contribution of sensitive features to be examined explicitly, rather than obscured within complex interactions and correlations. It is important to reemphasize that an EBM is not intended to recover causal relationships or the true data-generating process, but instead provides transparent predictions from the available data and features. Thus, including sensitive features can help better understand the structure of predictive signals present in the data. When used responsibly, transparent group-level importance analyses can help surface and scrutinize potential sources of bias, rather than inadvertently hiding them.

Conclusions

We propose a novel method to measure the importance of groups of features within Generalized Additive Models (GAMs) by computing their mean absolute joint contribution. This approach naturally extends the concept of individual feature importance in GAMs, and remains meaningful when there are many correlated features. Our definition is computationally low-cost, requires no model retraining, and yields similar results as groupSHAP in a real-world case study. We demonstrate its properties by applying a GAM-based Explainable Boosting Machine (EBM) to two real-world datasets (in neuroscience and healthcare) and three synthetic datasets. Our experiments reveal that group importance is particularly important for understanding the total signal in a group of correlated features, which can be overlooked when individual features are attributed low importance due to correlation. As such, group importance offers additional interpretability beyond individual feature importance in a wide range of applications.

Future work might involve applying group importances in real-world settings where feature selection can only be done at a group level. Secondly, the proposed metric of group importance assumes predefined groups, while methodology could be developed to discover these groups automatically, e.g. through hierarchical clustering using a group importance metric.

Author contributions

T.B. led the analysis, conducted neuroscience and synthetic experiments, and wrote initial draft of manuscript. L.S. and J.W. led the code development and integration into the InterpretML package. Y.W. and B.M. provided clinical healthcare expertise, and conducted the experiments on social determinants of health. J.D. prepared and analyzed the neuroscience data. O.K. and F.B. provided clinical neuroscience expertise. Q.Z. developed the fMRI segmentation methodology and led its application to the neuroscience experiments. R.C. conceived of the presented idea and developed the theory. K.M. collected and analyzed the neuroscience data. Both R.C. and K.M. provided guidance, were closely involved in writing the manuscript, and led the direction of the project. All authors contributed to discussion and reviewing the manuscript.

Funding

This work was generously supported by funding from the National Institute of Health (R01AA005965, R01MH145082, DA057567, U01AA017347, U24AA021697, K23AR082991), Stanford HAI Google Cloud Credits, HAI Hoffman-Yee Grant, the Innovator Grants Program by Stanford’s Department of Psychiatry, Microsoft’s AI for Good Lab, and Intelligible, Inc.

Data availability

Collection and distribution of data from the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA) were supported by NIH funding AA021681, AA021690, AA021691, AA021692, AA021695, AA021696, AA021697 and made available via the NIAAA Data Archive (collection 4513, https://nda.nih.gov/edit_collection.html?id=4513). The data for the real-world healthcare experiment is made publicly available by Pennsylvania Health Care Cost Containment Council (PHC4), and as of June 24, 2025, can be accessed at https://www.phc4.org/request-custom-data/available-data/. All data generated and analyzed for the synthetic experiments are included in this published article and its Supplementary Information file.

Declarations

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-43928-4.