Appendix 1—figure 9. Overestimation of model performance due to lack of data independence.

We trained new prediction models based on the k-fold cross-validation scheme, but without controlling for family structure. That is, for each testing participant, other members from the same family might be in training data. Compared with the k-fold results that controlled for family structure (Appendix 1—figure 8; dashed lines in this figure), results without controlling for family structure consistently overestimate model performance (average R² difference across regions: 0.9–2.1%; solid lines in this figure). This demonstrates the necessity of ensuring data independence between training and testing data to avoid biased model evaluations (see Varoquaux et al., 2017 for a similar issue with leave-one-trial-out). Specifically, training and testing data should not have members from the same family.