Normative Data Estimation in Neuropsychological Tests: A Systematic Review

Abstract

Objective

To quantify the evolution, impact, and importance of normative data (ND) calculation by identifying trends in the research literature and what approaches need improvement.

Methods

A PRISMA-guideline systematic review was performed on literature from 2000 to 2022 in PubMed, Pub-Psych, and Web of Science. Inclusion criteria included scientific articles about ND in neuropsychological tests with clear data analysis, published in any country, and written in English or Spanish. Cross-sectional and longitudinal studies were included. Bibliometric analysis was used to examine the growth, productivity, journal dispersion, and impact of the topic. VOSViewer compared keyword co-occurrence networks between 1952–1999 and 2000–2022.

Results

Four hundred twelve articles met inclusion and exclusion criteria. The most studied predictors were age, education, and sex. There were a greater number of studies/projects focusing on adults than children. The Verbal Fluency Test (12.7%) was the most studied test, and the most frequently used variable selection strategy was linear regression (49.5%). Regression-based approaches were widely used, whereas the traditional approach was still used. ND were presented mostly in percentiles (44.2%). Bibliometrics showed exponential growth in publications. Three journals (2.41%) were in the Core Zone. VOSViewer results showed small nodes, long distances, and four ND-related topics from 1952 to 1999, and there were larger nodes with short connections from 2000 to 2022, indicating topic spread.

Conclusions

Future studies should be conducted on children’s ND, and alternative statistical methods should be used over the widely used regression approaches to address limitations and support growth of the field.

Introduction

Clinical neuropsychology studies the relationship between behavior and brain dysfunction (Mitrushina et al., 2005). Neuropsychological assessment permits an understanding of patients’ behavior and underlying cognitive impairments (Kavé, 2005) through different tools such as neuropsychological tests (Mitrushina et al., 2005; Strauss et al., 2006). These tests must meet psychometric parameters, including validity, reliability, measurement error, sensitivity, specificity, and normative data (ND) (Arango-Lasprilla et al., 2017; Mitrushina et al., 2005; Strauss et al., 2006).

ND are reference values that help clinicians interpret test scores of individual patients compared to their peers (normative group) using sociodemographic variables (Innocenti et al., 2021). In other words, reference values provide an empirical context for test scores and a representation of the performance of a group (Mitrushina et al., 2005). Thus, ND allow the patient’s performance to be placed into categories (e.g., from very superior to impaired), which is necessary for important decision-making such as diagnoses (e.g., dementia; Mitrushina et al., 2005; Soer et al., 2009). Additionally, ND are useful for establishing therapeutic goals and evaluating intervention effectiveness. If a patient has below-expected cognitive performance, ND can serve as a guide for setting specific therapeutic goals and measuring the patient’s progress throughout treatment (i.e., Schneider & McGrew, 2018). Because of this, norms’ accuracy and representativeness of the population are important (Innocenti et al., 2021). Selecting appropriate ND will affect neuropsychological assessment result interpretation accuracy, reducing the probability of false diagnoses of cognitive impairment (Inesta et al., 2021).

Recent studies (Branco Lopes et al., 2021; Liozidou et al., 2021; Mascialino et al., 2022; Monette et al., 2023) showed a lack of ND for the most widely used neuropsychological tests throughout different countries (i.e., France, Greece, Ecuador, and Canada). This limitation leads neuropsychologists to use ND from other countries or to use raw scores. For example, more than 50% of clinicians in France (Branco Lopes et al., 2021), Canada (Monette et al., 2023), and Ecuador (Mascialino et al., 2022) and more than 30% of clinicians in Greece (Liozidou et al., 2021) reported using ND from other countries due to this limitation.

These practices may lead to serious errors during test selection, interpretation of the results, and resulting diagnosis. The use of direct scores to interpret results may be affected by external variables (e.g., sociodemographic variables) and create biases in interpretation (Rivera & Arango-Lasprilla, 2017; Van Der Elst et al., 2013). Likewise, the use of inadequate ND, either because they belong to another country, or because they do not represent other community members within the same country, might produce problems in the interpretation of results, as culture and sociodemographic characteristics (i.e., age, sex, and educational level) affect neuropsychological performance (Fernández & Abe, 2018; Rivera & Arango-Lasprilla, 2017). For example, it is well-known that ND change with respect to sample countries (Arango-Lasprilla et al., 2015; Arango-Lasprilla & Rivera, 2017; Rivera et al., 2019, 2021). Also, specific sociodemographic (e.g., living in a city or a rural environment, socioeconomic level) and cultural (e.g., acculturation level, language) factors could be important predictors for estimating ND.

Moreover, the Flynn effect, that intelligence quotient scores increase at the population level overtime, is also applicable in neuropsychological tests (Mitrushina et al., 2005; Strauss et al., 2006). For example, Iverson et al. (1999) demonstrated that patients with traumatic brain injury scored higher on the Controlled Oral Word Association Test than previously collected ND. Thus, reference values are more precise in the year they are created (Mitrushina et al., 2005). For this reason, the number of publications on ND in clinical neuropsychology has increased rapidly over time in part to meet demands for updated norms. Similarly, exclusion criteria for ND studies have been expanded over time when using healthy people. Thus, the samples used to generate norms are commonly highly educated or skilled, such that norms may overestimate scores and not be representative of the general population. ND for clinical samples (e.g., traumatic brain injury, Allen et al., 2012) have also increased overtime (Strauss et al., 2006).

The first study on ND published in the neuropsychology field was written by Coppinger and Ammons (1952) for the Full-Range Picture Vocabulary Test: VIII. In this study, age, sex, and the socioeconomic status of the children’s parents were used as relevant sociodemographic variables. Johnson et al. (1988) published ND for The Children’s Paced Auditory Serial Addition Task using not only the predictors age and sex, but also arithmetic, information processing, and intellectual ability.

These two studies are examples of the traditional approach to generate norms. However, according to Innocenti et al. (2021), two approaches exist: the traditional approach and the regression-based approach. The first approach divides the sample into subgroups based on some relevant demographic factors, such as age, education, and sex. Then, the mean () and standard deviation () are calculated within each subgroup. For the norming process, the samples are divided, making large jumps across groups, which causes low precision of the normative values (Innocenti et al., 2021; Van Breukelen & Vlaeyen, 2005). However, the regression-based approach is a relatively new method that allows the selection of important variables for a particular population through multiple regression. The multiple regressions obtain ND by the cumulative distribution of standardized residuals (Innocenti et al., 2021; Van Der Elst et al., 2013). Additionally, this method allows one to control for parameters, such as normality and multicollinearity (Variance Inflation Factor ≤ 10; Kutner et al., 2005) and identify possible influential cases. Furthermore, according to Innocenti et al. (2021) and Van der Elst et al. (2013), the regression-based approach has several additional advantages. Researchers and clinicians can use the entire sample to calculate ND, so the accuracy of the normative values is preserved. Additionally, the method allows one to see the significance and value of sociodemographic variables included in the model for that sample.

Even though the traditional approach is still used by some authors (Brunsdon et al., 2019; Lancaster et al., 2016), the regression-based approach has gained popularity (Timmerman et al., 2021). Due to the rapid increase in the number of publications related to ND in clinical neuropsychology, their impact and relevance for the field, and the increasing complexity of the regression method and calculations used to generate norms, the aim of this review is to quantify the evolution, impact, and importance of the calculation of ND over time. To our knowledge, this study presents the first literature review of methods for generating ND in neuropsychology. We believe it will help to identify what work has been done and what needs to be improved; it will therefore detect strengths and weaknesses in the field, and ND’s impact on clinical work. In this way, we hope that it will help to establish new methodology to create more accurate models in neuropsychology and help usher in new modeling approaches.

With these aims, a combination of PRISMA and bibliometric approaches will be used. PRISMA is based on a range of evidence for systematic reviews and meta-analyses and has traditionally been used to evaluate the effects of interventions. It can also be used as a basis for reporting systematic reviews (Page et al., 2021). A bibliometrics approach allows one to examine research activity and evolution in a given topic using diverse quantitative indices (e.g., doubling time, form of growth, Bradford zones, impact factor, etc.) that describe both the performance analysis of the general characteristics of publications and scientific mapping (Villalobos et al., 2022).

Methods

Systematic Review

Data source and search strategy

A systematic review was performed following PRISMA guidelines (Page et al., 2021) on April 12, 2022 using the following databases: PubMed, Pub Psych, and Web of Science (WOS) with a publication date limit equal to or greater than the year 2000. This cutoff date has been established because studies prior to 2000 only used the traditional approach (mean and standard deviation [SD]), with no variation in statistical analysis. The full line search strategy for the databases was: (“normative data”[Title] OR “normative”[Title] OR “norms”[Title]) AND (Neuropsychological test), and the following filters were applied in each database: “free full text,” “classical article,” “clinical study,” “English,” and “Spanish.”

Data Analysis and Categorization

Studies were chosen according to the following inclusion criteria: (1) were scientific articles; (2) were open access articles or be accessible from Public University of Navarre; (3) dealt with the topic of ND in healthy individuals or clinical populations in neuropsychological tests; (4) had good data-analysis development; (5) had a publication date equal to or greater than the year 2000; (6) were from any country; (7) were written in English or Spanish; and (8) were cross-sectional or longitudinal. The exclusion criteria were: (1) book chapters, systematic reviews or meta-analyses, conferences papers, or other forms of research that were not scientific articles (i.e., reports and unpublished manuscripts); (2) dealt with a topic other than the one proposed or only with ND or neuropsychological testing; and (3) failed to adequately describe methods or results.

The selection process was carried out as follows: (1) the search line was executed with the descriptors and filters described previously; (2) the bibliographic manager Zotero (Zotero desktop, version 6.0.1) was used to store the documents and citations and thus, facilitate the elimination of duplicate articles; and (3) the information of each study was stored in an Excel spreadsheet.

The results were analyzed chronologically by country of origin and keywords. Two researchers (AdC-T, LO-L) were responsible for reviewing each record in the Excel spreadsheet, whereas a second researcher (DR) was responsible for reviewing the procedure. The information summarized in this spreadsheet was transferred to Supplementary Material A with the relevant information: (1) first author and year; (2) country; (3) final predictors included; (4) ND for children or adults; (5) variable selection strategy; (6) approaches to generate norms; and (7) traditional versus regression-based approaches.

Bibliometric Analysis

Data analysis and categorization

For the bibliometric analysis, the same line search strategy for PRISMA was used in the PubMed database because PubMed encompassed the literature found in Pub Psych and WOS. A text file (“.txt”) was downloaded on April 12, 2022 with the keywords, abstracts, and authors of the article. This file was used in the “biblioshiny” interface through the “bibliometrix” package (Aria & Cuccurullo, 2017) from R Studio (version 4.1.3) to analyze the following bibliometric indices: price index, doubling time, annual growth rate, Bradford zones, and impact factor.

Price’s law (Price, 1963) is the most widely used indicator of the productivity and growth of a specific field or country (Villalobos et al., 2022). This productivity growth can be graphically synthetized by applying linear and exponential trend lines to a chronological distribution of the literature. Additionally, doubling time and annual growth rate were also calculated, as they are related to the growth of the field. To estimate the doubling time (D) of the scientific literature, the equation was used (Villalobos et al., 2022). The growth rate equation was (Egghe & Ravichandra Rao, 1992), where C(t) is the total number of papers produced at time t and c and g were the estimated constants of the data.

Bradford zones identify the specific discipline literature dispersion among journals and, thus, show which journals are selected by researchers to publish their results in a particular discipline (Bradford, 1934). Therefore, it provides important information to increase the likelihood of finding that bibliography.

Finally, the impact factor was used to evaluate the impact of a journal in a particular year and within a particular category (Garfield, 1979). It is calculated by dividing the number of citations in particular year by the total number of articles published in the two previous years (Garfield, 1979). This indicator is frequently used for ranking, evaluating, and comparing journals (Garfield, 1979; Villalobos et al., 2022). Although it is a controversial indicator (because it is calculated using as the numerator the total number of citations received by the journal, including news citations, editorials, and letters to the editor), it is still a benchmark index of the impact of journals (Villalobos et al., 2022).

Statistical Analyses

Statistical analyses were conducted using R Studio (version 4.1.3). The bibliometrix R package (Aria & Cuccurullo, 2017) was used to analyze bibliometric indicators. Also, linear and exponential trends from Price’s law were analyzed using R Studio (version 4.1.3). Finally, the free version 1.6.18 of the VOSViewer software was used to create, plot, and explore maps from bibliographic data (Van Eck & Waltman, 2010). It helps to visualize the state of the topic from the bibliography found in databases using keywords and their co-occurrence networks (e.g., Kuzior & Sira, 2022; Shah et al., 2019). For this study, two co-occurrence maps were created using the original PubMed strategy search line and inclusion/exclusion criteria: the first plots the keywords found in the publications from 1952 to 1999, and the second which plots the keywords from 2000 to 2021.

Results

Systematic Review

Data source and search strategy

The search strategy generated 900 results in PubMed, 2 in Pub Psych and 196 in WOS. The articles found in PubMed encompassed the literature found in Pub Psych and WOS, so 198 were labeled as duplicates.

After reading the publications’ titles and abstracts, 816 articles met eligible inclusion after discounting 198 duplicates. Four hundred and four articles were excluded because two were literature reviews, 53 were not open access or accessible through Public University of Navarre, 249 did not deal with ND in neuropsychological tests, 91 did not adequately report methods or results, and 9 were not scientific articles (i.e., reports and peer reviews). Finally, 412 were included in the systematic review. See Fig. 1 to visualize the schematic representation of the search and final result.

Fig. 1.

PRISMA flow diagram.

Findings

The results of the systematic review are presented, grouped by six topics: country, final predictors included, ND for children or adults, variable selection strategy, approaches to generate norms, and traditional versus regression-based approaches. Please see Supplementary Material A for a completed summary of the articles.

Country

Most of the studies (n = 378; 91.7%) developed ND for a single country (i.e., United States of America [USA], the Netherlands, and China). However, 33 (8%) of the studies generated ND for several countries simultaneously. Nevertheless, the country with the most ND for neuropsychological tests was the USA, followed by Italy and Spain.

Predictors

The most prevalent sociodemographic predictors associated with performance on neuropsychological tests were age, education, and sex. Age and education were usually represented as continuous (in years) or categorical variables (split up in age groups or by educational levels). Sex was dichotomized (male vs. female) in all studies. In addition, other studies have considered variables, such as language (Hall et al., 2018; Lindgren & Laine, 2007), race, or ethnicity (e.g., Beaumont et al., 2013; Belanger et al., 2022; Diehr et al., 2003; Gonzalez et al., 2006; Gonzalez et al., 2007; Walker et al., 2021), and diagnosis of a disease (e.g., Iverson et al., 2009), among others.

ND for children or adults

Most of the studies (n = 361; 87.6%) focused on adults or elderly populations, whereas ND for children and adolescents were less common (n = 39; 9.5%). The largest study of adults in the USA was Mayo’s Older African Americans Normative Studies (MOAANS; e.g., Ferman et al., 2005; Harris et al., 2002; Lucas et al., 2005). However, the NEURONORMA project (Aranciva et al., 2012; Calvo et al., 2013; Casals-Coll et al., 2013; Casals-Coll et al., 2014; García-Escobar et al., 2021; Palomo et al., 2013; Peña-Casanova et al., 2009) was the most well-represented for the Spanish-speaking adult population together with the Alegret study (Alegret et al., 2012) and NORMALATINA-Adults project published in a special issue by Arango-Lasprilla (Arango-Lasprilla, 2015; Guardia-Olmos et al., 2015). Within children’s studies in 2017, a group led by Arango-Lasprilla and Rivera conducted the NORMALATINA-Children project, which was the most relevant ND study for Spanish-speaking children (aged 6–17) from Latin America and Spain published in a special issue (Arango-Lasprilla & Rivera, 2017; Rivera & Arango-Lasprilla, 2017).

Neuropsychological tests

The neuropsychological tests with the most norms were the Verbal Fluency Test (n = 52; 12.6%), Trail Making Test (TMT; n = 48; 11.7%), Stroop Color Test (n = 31; 7.5%), Mini-Mental State Examination (MMSE; n = 24; 5.8%), Symbol Digit Modalities Test (SDMT; n = 23; 5.6%); Rey–Osterrieth Complex Figure (ROCF; n = 22; 5.3%), Boston Naming Test (BNT; n = 22; 5.3%), Montreal Cognitive Assessment (MoCA; n = 19; 4.6%), and Wisconsin Card Sorting Test (WCST; n = 18; 4.4%). All test scores were treated as continuous variables in the literature.

Variable selection strategy

The most common statistics used were linear regressions (204; 49.5%), Pearson’s, Spearman’s, bivariate correlation and/or covariance (n = 124; 30.1%), Analysis of variance (ANOVA), Multivariate analysis of variance (MANOVA), and/or Analysis of covariance (ANCOVA) (91; 22.1%), Student’s t and/or Mann–Whitney U (n = 55; 13.3%), and Chi-square (n = 9; 2.2%). However, 15 (3.6%) did not report any variable selection strategy.

Traditional versus regression-based approaches

Despite the fact that a majority of studies (n = 262, 63.6%) followed a regression-based approach, the traditional approach (n = 132; 32%) was also frequently used (e.g., Bozdemir & Gurvit, 2022), as well as other approaches (n = 13, 3.2%; e.g., Bayesian approach, fractional polynomial multiple regression, and LMS method). Moreover, five studies used both traditional and regression-based approaches (Bentvelzen et al., 2019; Bezdicek et al., 2012; Burggraaff et al., 2017; Llinas-Regla et al., 2013; Malisova et al., 2021). Between 2000 and 2005, both approaches were observed. For example, Armengol (2002), Campo and Morales (2003), and Lin et al. (2000) used SDs to calculate ND, whereas Laiacona et al. (2000), Norman et al. (2000), and Caffarra et al. (2002) used a multiple regression approach. However, in 2005, the number of ND articles following the regression-based approach increased due to the publication of a new regression method for ND generation by Van der Elst et al. (2005). At the same time, this method was being used with another type of nonneuropsychological test (Van Breukelen & Vlaeyen, 2005). The most relevant ND projects for both adults and children (i.e., MOAANS, NEURONORMA, NORMALATINA-Adults, and NORMALATINA-Children) used the regression-based approach.

Accordingly with the methodological strategy, in the traditional approach, most authors (n = 101; 75.4%) used means and SDs. However, in the regression-based approach, some variations were observed. Although linear regressions were used by most authors, quantile regressions (e.g., Vaughan et al., 2016) or logistic regressions (Gugssa et al., 2011) were used by others.

Presentation of ND

The way ND were presented differed across articles. Although the majority of authors used percentiles (n = 182; 44.2%), means and SDs (150; 36.4%), z-scores (n = 61; 14.8%), and T-scores (n = 34; 8.3%) were used too. Interestingly, Italian scientists established “equivalent scores” (e. g., Appollonio et al., 2005; Laiacona et al., 2000), where the scores range from 0 (representing scores below the cutoff point) to 4 (representing scores above the median; Caffarra et al., 2011).

With respect to specific research projects, MOAANS and NEURONORMA presented their normative values in percentiles and scaled scores, whereas for NORMALATINA-Adults and children projects, the reference values were presented in percentiles.

Bibliometrics

From the 2000 to 2022 period, 412 articles were retrieved using the same search strategy. Figure 2 shows the chronological distribution of publications, with a linear and exponential increasing number of articles over time. A linear curve explained 67.25% of variance, whereas an exponential curve explained 70.49%, suggesting that the increasing publication over time is fit better with exponential rather than linear growth. It seems that 2005 represents a cutoff given that after that date, the number of articles on this topic multiplied with respect to previous years. Similarly, one more cutoff point can be observed between 2018 and 2019 when the exponential growth exceeded a linear curve (Fig. 2).

Fig. 2.

Chronological distribution of the articles. Linear adjustment () appeared in gray, whereas exponential adjustment () in black. Note. : Variance.

The parameters obtained from the exponential model are shown in Table 1. The values of c and g were and , respectively. Therefore, the predicted growth of published articles can be handled by the equation from Egghe and Ravichandra Rao (1992), where . Accordingly, with this method, the literature about ND in neuropsychological tests has grown at a rate of 10.67% per year.

Table 1.

Parameters’ values obtained from the exponential model

Parameter estimates
Parameter	Estimate	Std.Error	95% Confidence interval
			Lower bond	Upper bond
c	0.055	0.026	−0.0005	0.110
g	1.260	0.205	0.828	1.692
ANOVA
Source	Sum of squares	df	Mean squares
Regression	1180.39	2	590.19
Residuals	474.60	18	26.36

Note: c and g are the estimated constants of the data. Std.error = standard error; df = degrees of freedom

Regarding the Bradford zones, the distribution of journals among zones can be seen in Table 2. Three journals (2.41%) were in Core Zone/Zone 1, whereas 7 (7.95%) and 78 (88.64%) were in Zones 2 and 3, respectively. Thus, most articles were published in a small core of specific journals: The Clinical Neuropsychologist, Archives of Clinical Neuropsychology, and Neurological Sciences. Table 3 shows the 10 journals with the highest number of publications on ND in the neuropsychological test’s topic. Most of the journals are at Q3, with an impact factor ranging from 1.971 (Applied Neuropsychology) to 5.486 (Neurologia).

Table 2.

Distribution of journals in Bradford’s zones

	N° of journals	% of journals	N° of articles	% of articles
Core/Zone 1°	3	3.41	169	41.02
Zone 2°	7	7.95	111	26.94
Zone 3°	78	88.64	132	32.04
Total	88	100.00	412	100.00

Note: N°= number; % = percentage.

Table 3.

Journals with the highest number of publications on normative data in neuropsychological test

Source	N° of documents	Total citations	Impact factor	Country of origin	Category	Quartile
Clinical Neuropsychologist	78	6.059	4.373	Netherlands	Psychology Clinical-SSCI	Q2
					Psychology-SCIE	Q1
					Clinical Neurology-SCIE	Q2
Archives of Clinical Neuropsychology	58	7.047	3.448	England	Psychology Clinical-SSCI	Q2
					Psychology-SCIE	Q2
Neurological Sciences	36	10.952	3.830	Italy	Neuroscience-SCIE	Q2
					Clinical Neurology-SCIE	Q2
Journal of Clinical and Experimental Neuropsychology	33	6.768	2.283	Netherlands	Psychology Clinical-SSCI	Q3
					Psychology-SCIE	Q3
					Clinical Neurology-SCIE	Q4
Neurorehabilitation	22	3.775	1.986	Ireland	Rehabilitation - SSCI	Q3
					Clinical Neurology - SCIE	Q4
					Rehabilitation - SCIE	Q3
Applied Neuropsychology. Adult	20	1.499	2.050	USA	Psychology-SCIE	Q4
					Clinical Neurology-SCIE	Q4
Journal of the International Neuropsychological Society	15	9.279	3.114	USA	Neuroscience-SCIE	Q3
					Psychology-SCIE	Q2
					Clinical Neurology-SCIE	Q3
					Psychiatry-SCIE	Q3
Neurologia	9	2.146	5.486	Spain	Clinical Neurology-SCIE	Q1
Aging Clinical and Experimental Research	6	8.102	4.481	Italy	Geriatrics & Gerontology - SCIE	Q2
Applied Neuropsychology*	6	703	1.971	USA	Psychology-SCIE	Q3
					Clinical Neurology-SCIE	Q3

Note: Data of JCR 2021; SSCI: Social Sciences; SCIE: Science; *Data of JCR 2013.

Figure 3 shows the percentage of publications in five of the most relevant countries for both traditional and regression-based approaches from 2000 to 2022, according to the affiliation of the corresponding author. The USA had the most publications (n = 96; 23.3%), and the regression-based approach was the most popular across countries.

Fig. 3.

Number of publications by country for each approach from 2000 to 2022.

Figures 4 and 5 show the connections between the keywords in two different periods: from 1952 to 1999 and from 2000 to 2022. In the first period (Fig. 4), small nodes with relatively long distance between them emerged, showing that there was a low frequency of occurrence. Moreover, four clusters emerged showing important themes related to the ND: top right, cognitive functions and psychometrics (i.e., references value, neuropsychological test, psychometrics, attention); central and top left, a combination of sociodemographic factors and cognitive functions (i.e., male, visual perception, intelligence); bottom, a combination of sociodemographic factors and analyses (i.e., regression analysis, educationa0l status); and to the right, reference standards.

Fig. 4.

Co-occurrence keywords map (1952–1999 period).

Fig. 5.

Co-occurrence keywords map (2000–2022 period).

In the second period (Fig. 5), a larger number of nodes with short interconnections appeared, suggesting that the ND calculation spread to many different areas. The largest node, bottom left, is mostly related to cognitive functions (i.e., space perception, verbal learning); the connections at the bottom right are associated with several disorders (i.e., multiple sclerosis, brain concussion); the cluster at the top left is related to language (i.e., speech, language tests); the cluster at the top right is related to the type of studies and samples (i.e., dementia, Parkinson disease); and finally, the cluster in the center is linked to neuropsychological tools (i.e., demographic, TMT).

Discussion

Due to the rapid increase of publication rate for ND in clinical neuropsychology, ND’s impact and relevance for the field, and the increasing complexity of the regression method and calculations used to generate norms, the aim of this review was to quantify the evolution, impact, and importance of the calculation of ND over time and identify current efforts and areas for improvement using systematic review (using PRISMA guidelines) and bibliometric approaches. To our knowledge, this study presents the first literature review of methods for generating ND in neuropsychology, and the results will allow researchers to improve modeling and norms generation methodology.

Systematic Review

The systematic review revealed that most studies (91.7%) developed ND for a single country rather than several countries simultaneously. It may be due to the time, economics, and personnel cost required to extract population data necessary for calculating ND (Mitrushina et al., 2005). Moreover, the country with the most ND for neuropsychological tests was the USA, followed by Italy and Spain. This may be due in part to the fact that USA employs a large number of neuropsychology researchers and has a much larger population than most European countries. This finding also may be due to the inclusion criteria for articles in the systematic review of having been written in English or Spanish. The review likely omitted many normative studies published in Asia or Africa in regional languages.

Common sociodemographic predictors associated with performance on neuropsychological tests were age, education, and sex. This is not surprising because these variables have historically been considered. Interestingly, other studies found other important variables beyond them associated with neuropsychological performance, including language (e.g., Collinson et al., 2014; Hall et al., 2018; Suarez et al., 2021), race/ethnicity (e.g., Casaletto et al., 2015; Kang et al., 2013; Lowery et al., 2004; Melikyan et al., 2021; Norman et al., 2000; Scarmeas et al., 2006), and disease status (e.g., Gifford et al., 2020; Iverson et al., 2009; Louey et al., 2014). These studies reflect the recurring issue in the field of neuropsychology and clinical assessment about the extent to which ND should be for a very specific population or whether it is better to have more diverse samples (e.g., Strauss et al., 2006).

One key finding from the systematic review was that most of the studies (87.6%) focused on adults or the elderly population, whereas ND for children and adolescents were less common (9.5%), highlighting a critical need for future pediatric normative studies. This is not surprising because neuropsychology began working with the adult population. Child neuropsychology is a relatively recent subdiscipline and is based on adult neuropsychology (Reynolds & Fletcher-Janzen, 2013). Moreover, collecting data from children and adolescents may be more complicated because the researcher must obtain permission from parents/guardians and/or institutions, and there are more variables to consider (e.g., parents’ education). Furthermore, if the target population is children under 6 years old, data collection is further complicated (Reynolds & Fletcher-Janzen, 2013).

The neuropsychological tests with the greatest number of normative studies were the Verbal Fluency Test, TMT, Stroop Color Test, MMSE, SDMT, ROCF, BNT, MoCA, and WCST. These tests are a central feature of many neuropsychological studies (Arango-Lasprilla et al., 2017; Branco Lopes et al., 2021; Monette et al., 2023), so it is appropriate that they are the best validated and normed in different populations. It is notable that all of these test scores were treated as continuous variables in the literature. Because of this continuous approach, it is similarly understandable that by far the most common statistical methods used were based on the general linear model, and chi-squares were used less than 3% of the time by comparison. In terms of open science and transparency, it may be an oversight of the neuropsychological literature that 3.6% of studies did not report how the variables were chosen.

Even though most studies (63.6%) followed the regression-based approach, the traditional means and SD approach (32%) was also common. In 2005, the number of ND articles following a regression-based approach increased likely due in part to Van der Elst et al. (2005) publication of a new regression method for ND generation for neuropsychological tests (in particular, the Rey’s verbal learning test). Some of the most relevant and impactful ND projects for both adults and children (i.e., MOAANS, NEURONORMA, NORMALATINA-Adults, and NORMALATINA-Children) used the regression-based approach, highlighting its importance in the field. In addition, widely used and referenced ND can be found in Test Manual format, such as Heaton et al., 2004, which generated regression-based ND for African American and Caucasian Adults from the USA. This switch of approaches could be due to the several advantages of the regression-based approach over the traditional approach. For one, regression does not split ND into smaller subsections like the traditional approach does (Innocenti et al., 2021; Van der Elst et al., 2011). In addition, it individualizes predicted scores and takes sociodemographic characteristics (e.g., age, gender, etc.) of each participant into account (Magnusdottir et al., 2021). A third advantage of the regression-based approach is that it can analyze unbalanced or biased data without biasing its estimation (Van der Elst et al., 2006). A final advantage is that the effect of age, level of education, and other covariates can be controlled for, so their impact on attrition does not bias ND (Appollonio et al., 2005; Laiacona et al., 2000; Van der Elst et al., 2008).

The way ND were reported differed throughout the articles with the majority of authors using percentiles (44.2%), means and SDs (36.4%), and z-scores (14.8%), though T-scores were used as well. This is understandable given the ease of interpretation by clinicians using these approaches. Other approaches such as “equivalent scores” (e.g., Appollonio et al., 2005; Laiacona et al., 2000) were less common, perhaps because there are mostly used in Italy, and some researchers have criticized them because equivalent scores are not able to differentiate between individual scores that are below the norm (e.g., Capitani & Laiacona, 1997). Some of the larger scale projects, like MOAANS and NEURONORMA, presented normative values in percentiles and scaled scores, or reference values in percentiles, as seen in the NORMALATINA-adults and children projects. Holistically, the use of these approaches points to several common methods for the calculation and use of normative values in neuropsychology.

Bibliometric Analysis

We explored the distribution of publications over time and found that the increasing trend in publications fit better with exponential rather than linear growth, suggesting that ND for neuropsychological tests has increased at an extremely rapid rate. An inflection point was identified at the year 2005, wherein the number of articles on ND for neuropsychological tests multiplicatively increased relative to previous years. This increase may be related to Van der Elst et al. (2005) new regression method publication, which resulted in an increasing use of the regression-based approach in the field. Similarly, another inflection point was between 2018 and 2019 when exponential growth exceeded a linear curve. This may be due to technological advances in terms of data collection and storage, as well as increased awareness of norms’ importance.

Bibliometric analysis also found that most articles were published in a core set of three neuropsychology and neurology journals. Most journals presented modest impact factors, suggesting the importance of ND within the field; however, the permeation of these data to other disciplines appeared somewhat limited. We found that the USA had the greatest number of publications. This might be due to the fact that the USA has a large population, many researchers from other countries published their papers with their USA’ affiliation (even if norms were conducted for other populations than the USA), as well as the inclusion criteria of incorporating only articles written in English or Spanish.

Co-occurrence networks in two different time frames (from 1952 to 1999 and 2000 to 2022) showed diverse picture of keywords connection. From 1952 to 1999, small nodes with long distances between them appeared, showing low occurrence of these keywords, which underlies the relatively small amount of ND for neuropsychological tests in this time period. Keywords could be clustered in five important topics related to ND: cognitive functions, sociodemographic factors, psychometrics, analysis, and reference values. A regression analysis node emerged, but regression was used to plot ND from the traditional approach rather than to calculate reference values as would be the case with the regression-based approach (Choynowski, 1970; Norman et al., 1975). In contrast, from 2000 to 2022, a greater number of nodes emerged with truncated interconnections between them, indicating that ND calculation had become quite widespread and was moving into different psychological subdisciplines. We also saw a prominent regression node, likely reflecting the exponential growth of ND generation and increased use of regression-based approaches during this timeframe.

Implications and Future Directions

One key finding from the systematic review was that most ND were collected for adults, highlighting a gap for future research to collect ND for children and adolescents. Another important finding was the increasing use of the regression-based approaches, which have several advantages over the traditional approach.

However, the regression-based approach does have important disadvantages. Specifically, it is not optimal when considering correlated test outcomes like scores on a multi-trial learning test (as is the case with ND) for three critical reasons. First, univariate analyses in regression do not use correlated individual trial scores, which can increase the precision of fixed effect parameter estimates in the model (Van der Elst et al., 2017). Second, fitting a regression model for each trial score independently can create several issues such as inflated Type I errors (Van der Elst et al., 2017). In a normative study, a Type I error means falsely rejecting the true null hypothesis that a variable such as age, gender, or years of education has no impact on the target test score. In effect, ND will be partitioned based on a nonrelevant demographic variable (Van der Elst et al., 2017); therefore, the reference values would not be accurate. In addition, age, education, and gender are relevant variables in neuropsychological assessment tests. Third, as (Van der Elst et al., 2017, p. 1175) note, “it is not parsimonious to use univariate statistical procedures.” For example, to generate ND for a multi-trial neuropsychological test, researchers would need as many regression equations as there are trials (Van der Elst et al., 2017).

Moreover, using a regression model requires several key assumptions that may not always hold with ND. According to Van der Elst et al. (2011), these assumptions include both those related to underlying assumptions of regression models and to extrapolating insights from regression models. Underlying assumptions of regression models include constant variance, independence, linearity, and an underlying normal distribution. The two most likely of these assumptions to fail with ND are linearity and an underlying normal distribution. The linearity assumption might fail if ND are not distributed linearly; instead, ND could have an exponential or cubic relationship, which is not well captured with a traditional linear regression. The underlying normal distribution assumption might fail if the ND are not distributed normally, as not all data conform to a normal distribution; in this case, a linear regression will likely not model the data well. The main assumption of extrapolating insights from regression models relevant to ND is that the relationship between demographic variables like age and test performance is the same across all values of the demographic variables, which may not be true. For example, the relationship between age and performance on neuropsychological tests might be different for young children than it is for adults; thus, resulting ND could become biased, and children with normal test results could be misclassified as being impaired, or vice versa (Van der Elst et al., 2011).

In addition, regression-based norms are based on the coefficients of the regression model, whereby the predicted score of a participant can be estimated according to his/her demographic characteristics. From this value, their residual is calculated, and the standardization of the residual is performed. Therefore, properly establishing a model that describes the performance of the population is the most important goal of the process.

Also, although a recent study showed that the regression approach requires a smaller sample size compared to the traditional approach (Oosterhuis et al., 2016), it is important to emphasize that both approaches require adequate sample size and representative sampling to avoid biases in the ND. Regarding the optimal sample size in the regression approach, it was not until Innoncenti’s (2023) publication that an equation and design (considering covariates/predictors) for the estimation of the optimal sample size as well as a robust design was proposed. This equation includes elements relevant to ND such as a confidence interval value (), number of predictors in the final model fit (), percentile value to considered “abnormal” (), and desired margin of error estimation (). Before this equation, a normative sample had to follow a similar distribution to the target population distribution (Strauss et al., 2006), making sample sizes large, especially at the top of the population pyramid.

Researchers are now able to, considering these weaknesses, implement new approaches in modeling. For example, the use of generalized linear models and generalized additive model, among others, may allow a better and more robust representation of the study population performance and, thus, have better reference values. These models can control for the random component and the relationship between the systematic component and the studied score.

Also, researchers could treat specific test scores, such as verbal fluency, as a Poisson rather than a normal distribution. In addition, researchers could use other discrete (binomial, negative binomial, etc.) and continuous (Gamma, Weibull, Beta, etc.) distributions to model ND. If researchers employ a regression model, one approach to improve performance could be to add splines to their models to allow the relationship between demographic variables like age and the test score to vary at different critical points, addressing the issue of extrapolating insights from these models. Similarly, if researchers notice the ND are distributed nonlinearly, they could add in restricted cubic or quadratic splines that can address any nonlinearity present in the data, although this approach complicates interpretability of coefficients.

Finally, to show the impact of having norms for a given population and clinical setting, examples are provided. Let us imagine a 49-year-old man from El Salvador with 14 years of education who completed the 15-Item Version of the BNT. Using ND from El Salvador (obtained from delCacho-Tena et al., 2023) this man would obtain a z-score of 0.02 and a percentile of 51. However, using the ND from Paraguay, this man would obtain a z-score of −1.66 and a percentile of 5. Thus, using norms from other population has severe implications because the man would go from an “Average” to a “Below Average” performance status (Guilmette et al., 2020).

However, it is not just the availability of norms that is important; the way these norms are created (using traditional vs. regression-based approach) also has clinical implications. To better understand this, another case is provided. Let us imagine an Argentinian man, aged 50 years, with 17 years of education and who needs 40 s to complete the TMT-A. With the traditional approach from the data provided by Fernández and Marcopulos (2008), this man would obtain a z-score of −0.68 and a percentile of 75, whereas with the regression-based approach from the data provided by (Arango-Lasprilla et al., 2015), this same man would obtain a z-score of −0.812 that corresponds to the 21st percentile. Now imagine an Argentinian man, aged 49 years, with 17 years of education and who need 40 s to complete TMT-A. With the traditional approach (Fernández & Marcopulos, 2008), this man would obtain a z-score of −0.41 and a percentile of 66, whereas with the regression-based approach (Arango-Lasprilla et al., 2015), this same man would obtain a z-score of −0.834 that corresponds to the 20th percentile. Note that with traditional approach, the percentile changes from 75th to 66th, whereas with regression-based approaches from 21st to the 20th percentile.

However, one may think that the differences noted previously may be simply due to sample differences, rather than the use of different methodologies. So, to conclude with the examples and demonstrate variations in ND depending on the approach, we will use the example given by Bentvelzen et al. (2019). These authors generated norms using both traditional and regression-based approaches but with the same sample. They proposed a fictional Australian man, 84 years of age and 13 years of education, with a modified-Telephone Interview for Cognitive Status score of 19. According to the traditional approach, this participant would have a z-score of −1.770 and, therefore, a percentile of 3.83, considered “Moderately Impaired.” However, using the norms-based regression approach, his z-score would be −1.150 and his percentile of 12.52, which is equivalent to “Low Average” (Bentvelzen et al., 2019). These examples demonstrate how can the performance of a patient change depending on the type of norms used.

Limitations

Despite the potential of the findings from this review to influence ND generation and use, there are several limitations that should be taken into account. First, although we recommend the use of other statistical models than just the regression-based or traditional approach, these approaches have yet to be widely used in practice with ND. Therefore, it is imperative that researchers make their use as straightforward as possible when likely many of the ultimate users of them will be clinicians who do not themselves have advanced statistical training. Second, although a multilingual (Spanish and English) approach was taken to identifying studies to be included in the current review, likely many studies were omitted, and as a result, the findings from this review cannot necessarily be generalized to global regions not represented here. Future studies should attempt to review articles published in other common languages (e.g., Mandarin, Hindi, French, Arabic, etc.). Additionally, future projects should expand the databases and consider scientific articles that are not only Open Access or accessible to Public University of Navarre. Moreover, for a more accurate analysis, bibliometrics should use indicators with a better fit, because in this study, the most common ones were used (e.g., impact factor, which is controversial).

Conclusion

This systematic review and bibliometric analysis considered 412 studies that met inclusion criteria. One important finding from the systematic review was that ND for neuropsychological tests for children and adolescents were far less common than for adults, illuminating a need for future studies to collect ND for children. Two key findings from the bibliometric review were that the amount of ND increased at an exponential rate and most recent studies used regression-based approaches. Due to the limitations of the regression-based approach, we conclude by arguing that future studies should employ other statistical approaches than regression-based approaches by treating specific test scores as Poisson distributions, when appropriate, and using other distributions than the normal distribution, including both discrete (binomial, negative binomial, etc.) and continuous (Gamma, Beta, T, Weibull, etc.) distributions to model ND.

Funding

Open access funding provided by Universidad Pública de Navarra

Conflict of interest

None declared.

Acknowledgements

We acknowledge the support of the Public University of Navarre.

Authors' contributions

Ana delCacho-Tena (Data curation, Formal analysis, Methodology, Resources, Software, Visualization, Writing—original draft), Bryan R. Christ (Resources, Writing—original draft), Juan Carlos Arango-Lasprilla (Conceptualization, Writing—review & editing), Paul B. Perrin (Supervision, Writing—review & editing), Diego Rivera (Conceptualization, Data curation, Supervision, Writing—review & editing), and Laiene Olabarrieta-Landa (Conceptualization, Data curation, Supervision, Writing—review & editing).

Data availability statement

The data underlying this article are available in the article and in its online supplementary material.