Harmonizing ethno-regionally diverse datasets to advance the global epidemiology of dementia

Introduction

Age is the biggest risk factor for cognitive impairment (CI) and dementia, and the global societal and financial burdens these conditions impose are rising as the world’s population ages.¹ Globally, the number of people with dementia is estimated to reach around 150 million by 2050, with the greatest increases expected to occur in developing regions, including Africa.¹ Research on CI and dementia is lacking in many low- and middle-income countries (LMICs).² However, the research results from one country or population do not apply to another, with reported differences in the epidemiology of CI and dementia between countries,³ as well as between different races/ethnicities within countries.⁴ Research into how cognitive decline can be slowed and CI and dementia ultimately prevented are thus necessarily a global effort, using large samples with data from different ethno-regions. Resource and co-ordination limitations mean that data on this scale will typically not come from a single source. Rather, such data must be collated from across multiple unique sources focused on particular countries or regions.

The data needed to understand the epidemiology, etiology, and risk and protective factors for CI and dementia comprise a vast array of types, including, but not limited to, demographics, diagnoses, cognitive or neuropsychological test results, medical histories, lifestyle variables like physical activity, substance use and diet, functional status, neuroimaging, and biomarkers. Each data type can be assessed in many ways, and collaborative efforts that use data from multiple sources are faced with the challenge of making these data comparable so they can be pooled for analysis or more accurately compared.

In this review we discuss how data used in dementia and cognitive impairment research can be made more comparable by harmonization. We cover the benefits and challenges of harmonization, and outline broad retrospective and prospective approaches. We also describe harmonization for particular data types, focusing on neuropsychological test results and neuroimaging, but also including dementia diagnoses, behavioral and psychological symptoms of dementia instruments, and electroencephalography measures.

Discussion

What is data harmonization? Qualitative and quantitative approaches

Harmonization is the process by which data for similar measures or constructs from different sources are made more comparable, or inferentially equivalent.⁵ The type of harmonization process needed to achieve comparability depends upon the sort of data involved, and may be qualitative or quantitative.

Qualitative approaches lead to data from different sources having a common format, such as the same range of response options or categories, sometimes requiring a transformation process.^6,7 Examples of this approach include:

• Choosing an item from each source that best represents the measure or construct of interest, e.g., different questions addressing subjective cognitive decline (for a more detailed account, see Box 1)

• Creating a categorical variable by choosing cut-points for different continuous scales measuring the same construct, e.g., classifying the presence of current depression based on a score of 6+ on the Geriatric Depression Scale (GDS) used by one source, and a score of 16+ on the Centre for Epidemiological Studies depression scale (CES-D) used by another source (for a more detailed account see Table S11 in Lipnicki et al.⁸)

• Collapsing response categories in the data for some sources to make them similar to those for data from another source with fewer response categories, e.g., self-rated health scales with different numbers of response categories (see Table 1).

Box 1. Qualitative harmonization of self-experienced decline in cognitive capacity.

Subjective cognitive decline is self-experienced decline in cognitive ability from a normal level in the absence of objective impairment, and may be the first sign of Alzheimer's disease.⁶⁴ A recent collaborative research project aimed to estimate the prevalence of subjective cognitive decline (SCD) in and across international cohort studies of aging.⁶⁵ Each study contributing data to the project asked their participants different sets and numbers of questions relevant to determining self-experienced decline in cognitive capacity, requiring the data to be harmonized for more accurate comparison and pooling.

The project used two approaches to harmonizing self-experienced decline in cognitive capacity: qualitative and quantitative.

Qualitative: Two authors independently compared all items assessing self-experienced decline in cognitive capacity across the studies, and identified one common item from each that broadly addressed problems or difficulties with memory. The original data for these items were transformed to a binary variable indicating the presence or absence of self-experienced decline in cognitive capacity, with any indication of decline in the original responses categorized as “presence”.

Study	Item selected for qualitative harmonization	Original coding
Active Ageing	Do you feel you have more problems with your memory than most?	1 = yes, 2 = no
CFAS	Have you ever had any difficulty with your memory? If yes, is that a problem for you?	0 = no, 1 = yes, moderate, 2 = yes, severe
EAS	Compared with one year ago, do you have trouble remembering things more often, less often or about the same?	1 = more often, 2 = less often, 3 = about the same
SLASII	Overall, how would you rate your memory or other mental abilities as compared to earlier period of your life (more than one year ago)?	1 = much better, 2 = a bit better, 3 = a bit worse, 5 = much worse

Note: only 4 of the 16 studies included in the project are shown.

Table 1.

Example of self-rated health scale harmonization

Study	Coding of original response options to Very good = 1, Good = 2, Poor = 3
Bambui	Very Good, Good = 1; Reasonable = 2; Fair = 3
CFAS	Excellent = 1; Good = 2; Fair, Poor = 3
EAS	Excellent, Very Good = 1; Good = 2; Fair, Poor = 3
HK-MAPS	Cumulative Illness Rating Scale sum of various organ system severity ratings: 0,1=1; 2-4=2; 5-13=3
Invece.Ab	Visual analogue scale: 0-6 = 1; 7-8 = 2; 9-10 = 1
KLOSCAD	Excellent, Good = 1; Fair = 2; Poor = 3
LEILA75+	Very Good/Excellent = 1; Good = 1; Fair = 2; Poor = 3; Very Poor = 3
MoVIES	Excellent = 1; Good = 2; Fair, Poor = 3
PATH	Excellent, Very Good = 1; Good = 2; Fair, Poor = 3
SALSA	Excellent, Very Good = 1; Good = 2; Fair, Poor = 3
SGS	Very Good = 1; Good = 2; Fair, Poor = 3
SLASI	Excellent, Very Good = 1; Good = 2; Fair, Poor = 3
Sydney MAS	Excellent, Very Good = 1; Good = 2; Fair, Poor = 3

Note. Taken from S9 Table in Lipnicki et al. showing how the original self-reported health data from 13 international cohort studies of aging were harmonized to a 3-category variable representing response options very good, good, and poor. (Adapted from Lipnicki DM, Makkar SR, Crawford JD, et al. Determinants of cognitive performance and decline in 20 diverse ethno-regional groups: A COSMIC collaboration cohort study. PLoS Med. 2019;16(7):e1002853; under CC BY 4.0)

Quantitative harmonization is needed for more complex data types and often requires statistical processing to bring them to a common format.⁹ Statistical harmonization is typically required for data from cognitive or neuropsychological tests, of which there are hundreds that differ on characteristics like the particular cognitive abilities assessed, and the depth to and mode by which they are assessed.¹⁰ A detailed account of approaches to the statistical harmonization of neuropsychological test scores is given in a later section.

Benefits of harmonized data

Harmonization is an often-necessary step before integrative data analysis, in which individual participant level data from multiple sources are analyzed simultaneously. Integrative data analysis techniques such as mega-analysis and individual participant data meta-analyses can overcome some of the limitations associated with single studies or meta-analyses of aggregated study data.^6,9 The benefits of harmonized data thus include the capacity to:

• Pool data from different sources, which increases the sample size and thereby the statistical power:

  • This is particularly important when analyzing rare conditions, characteristics, or outcomes, given the increased absolute numbers of individuals with these (for details see Hussong et al.⁶)
  • Pooling data can similarly increase the number of participants from subgroups that may be typically underrepresented in single studies.⁶
• Make more accurate comparisons across data sources using measures that are more similar:

  • This is particularly relevant for investigations of commonalities and differences in factors contributing to CI and dementia across different countries, regions of different economic development, or different races/ethnicities¹¹ (for examples of relevant research studies see Box 2).

• Conduct validation of results or replication across multiple data sources.⁵

Box 2. Using harmonized cognitive impairment and dementia data for international comparisons.

When researching MCI and dementia on a global scale, a great benefit of using harmonized data is the capacity for more accurate comparisons across different ethno-regions.

• More accurate comparisons of prevalence and incidence of dementia and related conditions. While the high variation in reported rates of mild cognitive impairment (MCI) across different countries are partially explained by differences in location and demographics, there is a significant contribution from differences in definition and methodology.⁶⁶ These differences can be reduced by harmonizing cognitive test, functional and subjective cognitive complaint data and applying a uniform approach to classifying MCI. This approach has yielded much more similar rates of MCI than previously reported.²⁵ The figure shows the prevalence of MCI previously reported for seven cohort studies representing five different countries alongside more uniform rates produced using harmonized data.²⁵

• Better understanding of risk factors for dementia and related conditions as universal, or as differing between races/ethnicities and regions, including strength of association between risk factors and outcome. Not only are there ethno-regional differences in the prevalence of risk factors for dementia, such as more diabetes and hypertension in developing countries like India,² but analysis of harmonized data on an international scale suggests that the strength of association between particular risk factors and CI and dementia can also differ.⁸ A risk factor’s prevalence and strength of association with dementia determine the proportion of dementia in a population that can be attributed to the risk factor. This proportion was able to be estimated for various dementia risk factors and more accurately compared across eight countries where identical 10/66 protocols had been used.⁶⁷

Other benefits associated with harmonization include the opportunity for extended use of existing datasets through collaborative projects where data are shared.⁷ Indeed, data sharing has become of increasing importance, with many publishers and funders now encouraging or requiring data sharing, for example, the publishing company Elsevier¹² and the National Institutes of Health, USA.¹³

General challenges of data harmonization

The potential for different data sources to have used considerably different methods to measure the same construct often makes harmonizing data challenging, particularly for cognitive data, given the vast range of tests available.¹⁰ The process can be time consuming and resource intensive,⁵ even more so when done on a global scale where translation and cultural differences may need to be considered. It should also be noted that harmonization is often specific to the requirements of a certain research question.⁵ Further, transformation of raw data to harmonized data can involve some loss or distortion of information, such as when a variable with five response options or a continuous scale is collapsed to a common format variable with three response categories (see Table 1).

Retrospective and prospective approaches to data harmonization

Most of our discussion of harmonization refers to retrospective data harmonization, which is applied to pre-existing data where constructs or characteristics of interest were obtained or recorded differently by different sources (for an example see Box 1). An alternative approach is prospective data harmonization, which is the implementation of uniform protocols across different studies or research centers before data collection occurs, so that data are collected in a harmonized way. Examples of prospective data harmonization on an international scale are the 10/66 dementia research group protocols for addressing dementia epidemiology in Latin America, China and India,¹⁴ the Harmonized Cognitive Assessment Protocol designed to enhance comparisons across international sister studies of the U.S. Health and Retirement Study,¹⁵ and the Latin America and the Caribbean Consortium on Dementia (LAC-CD), which aims to facilitate comparisons of dementia between countries with harmonized dementia diagnoses.¹⁶ Similarly prescriptive approaches have been developed for retrospective data harmonization, including a set of guidelines outlining the procedural steps.⁵ There have also been attempts to develop systems that facilitate retrospective data harmonization, such as DataSHaPER¹⁷ and the BioSHaRE Project.¹⁸ Full adherence to a harmonized protocol can be compromised by context-dependent requirements, such as the need to replace a cognitive task requiring spelling ability in populations with low rates of literacy.¹⁹ In addition, it has been suggested that the evidence produced by repeated implementation of a protocol across samples may be weaker than evidence from studies using different methodologies.⁵

Harmonizing neuropsychological test data

Neuropsychological test data can be complex to harmonize. There are more than 500 neuropsychological tests²⁰ and 70 different tests commonly used to assess dementia.²¹ A simple method to harmonize such data is to analyze a common test or set of tests and treat raw scores as equivalent across sources. In aging research, this has been done for a limited number of widely used measures, like the Mini-Mental State Examination (MMSE).²² However, this approach excludes potentially useful studies that do not use the same test(s) as others. Also, when evaluating cognition across different ethno-racial populations, it is traditional to base assessments on standardized scores (using appropriate norms) rather than regard raw scores as being equivalent. When different sources use different tests, harmonization requires a statistical approach, of which there are three broad methods:^9,23

• Standardization

• Latent variable modelling

• Use of multiple imputation.

Standardization

Standardized scores can be used to interpret an individual’s test performance. Some test manuals present standardized scores (z-scores with a mean of 0 and a standard deviation [SD] of 1) for different demographic groups, defined by sex and ranges of age and/or education. These demographically adjusted standardized scores are the ones most commonly used when neuropsychologists determine diagnoses of Mild Cognitive Impairment (MCI) or dementia. However, when harmonizing test scores across studies from different ethno-racial populations, such manuals are usually not available. In this situation, regression models have been used to produce demographically adjusted standardized scores, using an appropriate normative sample. In community based longitudinal studies, the baseline sample (excluding those with serious illness or dementia) has been used as the normative sample. Demographically adjusted scores can then be obtained as the standardized residuals in regression models, with demographic variables (usually age, sex and education) as the independent variables and the raw test scores as the dependent variable. Equations used to obtain these standardized scores at baseline can then be applied to raw scores at later waves, to produce scores which are comparable across waves. This method of harmonizing cognitive tests across cohorts has been done previoulsy^24,25 (see Box 2 for an illustration on harmonizing MCI diagnoses based on standardized cognitive scores).

When research examines the associations of age, sex and education with cognitive performance, demographically adjusted scores would not serve as the appropriate outcome variables. If analyses are confined to a single study, z-scores with means and SDs calculated using the baseline sample (or other appropriate normative sample) could be used. However, such within-study z-scores would not be comparable across studies owing to their different distribution of demographic characteristics.

One solution is to form “demographic category-centred scores” (or C-scores).^9,26 Here, subsamples with the same sex and ranges of age and/or education are selected in each study, and their means and SDs are used to calculate C-scores within each study. For example, subsamples of women aged 70—74 years with 8—13 years of education were used to harmonize cognitive test scores in three Canadian studies.²⁶ A limitation of this method is the possibility of not obtaining subsamples of sufficient size to reliably estimate the means and SDs required. To overcome this, a modified procedure uses regression models to estimate means and SDs withing each study, conditional on common values of the demographics, chosen to be close to the mean or median values across all studies.²⁷

Latent variable modelling

Latent variable modelling assumes the existence of latent factors (or constructs) underlying a set of neuropsychological tests or test items (or more generally, observed indicators). Two modelling methods are the use of Item Response Theory (IRT) based models and Linear Factor Analysis (LFA).

IRT is a framework for understanding the psychometric properties of a test and its items.^28,29 IRT is especially relevant in integrative data analysis,^30,31 because it allows the identification of item biases across studies and demographic groupings, referred to as differential Item functioning (DIF), and it uses tests that are both common and noncommon across studies to estimate the underlying construct (for an illustration of linking see Box 3). IRT-based latent variable modelling has been used for harmonizing longitudinal cognitive data.³² An example of employing LFA in structural equation modelling to obtain latent cognitive factors can be found in Salthouse et al.³³

Box 3. Quantitative harmonization of self-experienced decline in cognitive capacity.

The quantitative harmonization approach used both common and unique items to model the latent construct of self-experienced cognitive decline that is equivalent in meaning and metrics across studies.⁶⁴ The common item serves as an anchor to link the unique items, for example, item 2 in Study 2 can be linked to item 6 in Study 3 via the common item. The 2-Parameter Logistic (2-PL) Item Response Theory (IRT) model^30,31 was used to evaluate measurement equivalence of the items (item difficulty and item discrimination) across studies, and based on the model, latent scores for each participant were estimated.

	Study
Items	1	2	3	4
1. Common item (see Box 1)
2. Have you tended to forget things recently?
3. Difficulty remembering names/things of close people
4. Difficulty remembering where you kept/put things
5. More effort to remember things than used to?
6. In the past year, how often did you have trouble remembering things?
7. Memory worse than 10 years ago

Recently, a Moderated Nonlinear Factor Analysis (MNLFA) model has been developed to handle mixed distributions of observed indicators (e.g., binary, ordinal and continuous).³⁰ This method has the additional advantages of modelling non-linear associations between items and the latent factor, and allowing the model parameters to be moderated by categorical (e.g., sex, study membership) and continuous (e.g., age) covariates simultaneously for testing DIF.

Multiple Imputation

Tests or test items that are not assessed in a particular study can be considered as missing by design, and handled using statistical models like multiple imputation. Values for missing items/tests in one study can be imputed using information from items/tests overlapping across studies as well as other related variables in the combined data set, but does not require the overlapping items/tests to be in every study. Typically, multiple imputed data sets are generated, and each analysed separately en route to a pooled estimate. Alternatively, values can be averaged across the imputed data sets to generate a full data set. Burns et al.³⁴ shows how missing MMSE item scores across studies can be imputed and a full data set analyzed.

Harmonizing neuroimaging data

Magnetic resonance imaging (MRI) data can be valuable for understanding and diagnosing neurodegenerative diseases.³⁵ The cost and time associated with collecting neuroimaging data mean it is often necessary to combine data collected from multiple sites and across diverse populations and experimental conditions to enhance both statistical power and generalisability of findings. This multisite approach to the collection and analysis of neuroimaging data for dementia research includes the Alzheimer’s Disease Neuroimaging Initiative (ADNI),³⁶ ENIGMA³⁷ and CHARGE³⁸ consortia. A major challenge for pooling multi-site neuroimaging is the lack of standardisation in both technical aspects (i.e., scanner platforms, image acquisition and processing protocols), as well as differences in sample characteristics (i.e., inclusion/exclusion criteria and sample size).³⁹

Methods for the prospective harmonisation of neuroimaging data in the dementias field have been developed by consortia, multi-centre studies and working groups and can include standardisation of: definitions and frameworks (e.g., for imaging of white matter hyperintensities⁴⁰), imaging acquisition protocols (e.g., for vascular dementia⁴¹) and segmentation procedures (e.g., for hippocampal volume⁴²). Data quality control procedures can also be standardised,⁴³ while containerised software packages can be distributed to ensure consistency in software across sites and time.⁴⁴ However, studies have shown that even after careful prospective harmonisation, systematic differences in images and sample characteristics across sites may lead to bias in MRI-derived measures.⁴⁵ Retrospective data harmonisation approaches have therefore been developed that allow the pooling of imaging datasets from heterogeneous sources in an unbiased manner.

One of the most widely used methods for retrospective harmonisation of neuroimaging data is the ComBaT approach, a technique originally developed to remove batch effects in genomics data.⁴⁶ ComBaT was first extended to the harmonisation of diffusion tensor imaging data,⁴⁶ and has recently been applied to the harmonization of structural neuroimaging data in both cross-sectional³⁹ and longitudinal contexts,⁴⁷ as well as functional neuroimaging data.⁴⁸ ComBat corrects for site (or scanner) differences via an empirical Bayes algorithm that estimates and removes location (mean) and scale (variance) differences across sites prior to downstream analysis. Clinically-relevant variations are preserved by defining covariates of interest and incorporating their effect on the variance. ComBAT has been applied to the harmonisation of dementia datasets⁴⁹ and shown to outperform other site correction techniques.³⁹

Other approaches to the harmonisation of multi-site neuroimaging data include Neuroharmony, a supervised machine learning approach that predicts ComBaT correction factors from imaging quality metrics.⁵⁰ In a process akin to pediatric growth charts, normative modelling uses percentiles to chart the variation of an outcome brain measure normed to the variation of a set of clinically-relevant covariates which, in a multisite framework, can include site as a covariate of interest.⁵¹ Recent reviews have identified the potential of this normative approach to address heterogeneity in neuroimaging models of dementia.⁵² Deep learning approaches have also been developed that are based on generative adversarial networks. These aim to extract a set of imaging features that are maximally informative for an outcome of interest (e.g., Alzheimer’s disease) while also being maximally uninformative about the site or scanner where the data originated.⁵³ These approaches to the retrospective harmonisation of neuroimaging data have their advantages and disadvantages,⁵⁴ but each has the potential to provide more powerful and generalisable research into neurodegenerative disorders.

Harmonizing dementia diagnoses

Autopsy-based diagnoses are the gold-standard for dementia and other neurodegenerative diseases. Recent advancements in brain imaging, such as positron emission tomography (PET) scans for amyloid beta and tau, have improved the accuracy of Alzheimer’s disease diagnoses. However, this is expensive and not always possible for cohort studies of aging, especially in LMICs. Many research studies therefore rely on clinical diagnoses of dementia, but there are substantial differences in diagnostic procedures (e.g., consensus by an expert panel, assessment tools like the Clinical Dementia Rating scale, the Geriatric Mental State interview) and criteria (e.g., DSM-III-R, DSM-IV, ICD-10) across studies.^1,55 These methodological differences can result in varying estimations of dementia rates.⁵⁶ Dementia can be diagnosed from assessments of cognitive performance and instrumental functioning, and algorithms derived from these can be a standardized method of dementia classification across studies (see Prince et al.⁵⁷ for an algorithm developed in the 10/66 project). Recently, an IRT based model was used to harmonize dementia classifications in two cross-sectional studies,⁵⁸ but its application to a larger number of and more diverse studies has yet to be examined.

Harmonizing behavioral and psychological symptoms of dementia (BPSD) instruments

One challenge for the collection and pooling of BPSD data across studies is the large array of available tools that measure the same or similar constructs. In terms of prospective harmonization of BPSD measures, several consensus guidelines have been developed,⁵⁹ with many recommending the Neuropsychiatric Inventory for global assessment of BPSD, as well as more specific measures such as the Geriatric Depression Scale and the Dimension Apathy Scale.⁵⁹ Many of these recommended tools are available in multiple languages, including those from LMICs.

Quantitative approaches to the retrospective harmonization of BPSD measures also hold great promise for pooling data that have already been collected or when the adoption of consensus guidelines is not appropriate. Harmonization across BPSD measures often necessitates the identification of common items for linking purposes, and this process for BPSD measures has been detailed recently in a reproducible manner.⁶⁰ However, when compared with quantitative harmonization of cognitive measures, the application of these approaches to BPSD instruments has been limited.

Quantitative harmonization has been used to develop common metrics, or crosswalks, which link various measures of neuropsychiatric symptoms,⁶¹ though this approach has not yet been initiated in the dementias field.

Harmonizing electroencephalography (EEG) measures

As a low cost and minimally invasive measure of brain connectivity, EEG represents a viable option for measuring dementia biomarkers in LMICs. To encourage multicentre harmonization of EEG data, the Electrophysiology Professional Interest Area and Global Brain Consortium have endorsed recommendations for EEG measures in clinical trials of Alzheimer’s Disease, including for stratification of participants and the monitoring of disease progression.⁶² Meanwhile, recent efforts have focused on developing standardised guidelines and best practices for EEG data acquisition, preprocessing and data analysis that can be applied to multicentre EEG studies of brain connectivity more broadly.⁶³

Summary

Dementia research is enhanced by bringing together data from multiple sources. However, methodological heterogeneity means that the data typically need to be retrospectively harmonized, sometimes even when prospective approaches to minimize heterogeneity have been implemented. The particular harmonization methods required depend on the data type, and range from a relatively simple choice of comparable items across sources, to the statistical and technology-driven methods needed to harmonize neuropsychological test scores and neuroimaging data, respectively. While often a resource intensive process, harmonization can facilitate data pooling and thereby enhance statistical power. Harmonization can also enable more accurate comparisons, such as comparisons of the prevalence and effects of risk factors for dementia across diverse ethno-regional groups.

Key Points:

• Data from multiple sources often represent heterogeneous methodology that includes different assessment instruments and classification criteria.

• Harmonization is the process by which data for similar measures or constructs from different sources are made more comparable.

• Harmonization enables data from multiple sources to be analyzed simultaneously, with techniques such as mega-analysis and individual participant data meta-analyses.

• Statistical harmonization is needed for neuropsychological test data, with methods including standardization, latent variable modelling, and the use of multiple imputation.

• The most popular approach for harmonizing neuroimaging data is ComBaT, with other applications to dementia research including normative modelling and machine learning approaches to statistical harmonization.

Synopsis:

Understanding dementia and cognitive impairment is a global effort needing data from multiple sources across diverse ethno-regional groups. Methodological heterogeneity means that these data often require harmonization to make them comparable before analysis. We discuss the benefits and challenges of harmonization, both retrospective and prospective, broadly and with a focus on data types that require particular sorts of approaches, including neuropsychological test scores and neuroimaging data. Throughout our discussion we illustrate general principles and give examples of specific approaches in the context of contemporary research in dementia and cognitive impairment from around the world.

Clinical care points.

• With the increasing digitalization of medical care, data from diverse sources must be harmonized for efficient clinical care and facilitation of clinical research.

• Barriers and facilitators of harmonization should be identified at the national and international levels, so that global clinical research and practice can inform clinical care and prevention of dementia in all jurisdictions.

• Policies and frameworks should be put into place to facilitate harmonization of clinical and research data at both national and international levels.