Cross-cultural neuropsychological assessment in the European Union: a Delphi expert study

Abstract

Objective

The increasing ethnic diversity in the European Union (EU) calls for adaptations to neuropsychological assessment practices. The aims of this study were to examine the current state of cross-cultural neuropsychological assessment in EU-15 countries and to provide recommendations for researchers and policy makers.

Method

Twelve experts from nine EU-15 countries participated in a Delphi consensus study involving two sequential rounds of web-based questionnaires and an in-person consensus meeting. The experts individually rated Delphi topics on the basis of importance (scale 1–10). The degree of consensus was determined by assessing first and third quartiles (Q1 and Q3) and medians.

Results

Consensus outcomes showed the following priorities: (a) the development of tests (median importance rating 10, Q1-Q3: 9–10), (b) the collection of normative data (median importance rating 9, Q1-Q3: 8–10), and (c) more training, awareness, and knowledge regarding cross-cultural assessment among neuropsychologists in the EU (median importance rating 9, Q1-Q3: 8–10). Whereas memory tests were often available, tests measuring social cognition (median 9, Q1-Q3: 8–10) and language (median 9, Q1-Q3: 7–10) are particularly lacking. Recommendations were made regarding essential skills and knowledge necessary for cross-cultural neuropsychological assessment.

Conclusions

This study in a small group of experts suggests that the development and availability of cross-cultural tests and normative data should be prioritized, as well as the development and implementation of training initiatives. Furthermore, EU guidelines could be established for working with interpreters during neuropsychological assessment. Before implementing these recommendations, follow-up studies are recommended that include more minority neuropsychologists and community stakeholders.

Introduction

Although a certain degree of diversity has always been present in European Union (EU) countries, diversity levels have increased greatly over the last seven decades, starting with the immigration of labor workers from countries outside the EU from 1950–1974 and the immigration of people from once colonized countries, followed by the influx of asylum seekers and refugees in more recent years (van Mol & de Valk, 2016). Therefore, the 15 original EU-countries, or EU-15, have had to adjust rapidly to the increasing diversity in their societies. Several minority ethnic groups in EU-15 countries are at an increased risk of medical conditions that are associated with cognitive impairment, such as stroke (Kunst, Stronks, & Agyemang, 2011), diabetes mellitus (Kunst, Stronks, & Agyemang, 2011), and dementia (Parlevliet et al., 2016; Rosenbaum, Kristensen, & Schmidt, 2008). Furthermore, other conditions that can influence cognition may occur in some minority ethnic groups, such as tropical diseases like malaria (Monge-Maillo & López-Vélez, 2012) and schistosomiasis (Lingscheid et al. 2017) in people who recently emigrated from endemic areas, malnutrition in refugees (Kristiansen, Razum, Tezcan-Güntekin, & Krasnik, 2016), and exposure to occupational hazards, such as pesticides, in labor workers (Parrón, Hernández, Pla & Villanueva, 1996). As a result, neuropsychologists in EU-15 countries will increasingly encounter patients from minority ethnic groups in their daily practice.

Several characteristics of minority ethnic groups may pose unique challenges to neuropsychologists. First, limited proficiency in the host country language is widespread among older people in some minority ethnic groups in EU-15 countries, including Moroccans and Turks in the Netherlands (Schellingerhout, 2004), South Asians in the UK (Blakemore et al., 2018), Turks in Germany (Van Tubergen & Kalmijn, 2005), and Turks and Vietnamese in Belgium (Van Tubergen & Kalmijn, 2005). The language in which neuropsychological tests are administered, as well as the level of formality used, can significantly impact communication, rapport, and subsequent test scores (Ardila, 2007a; Boone, Victor, Wen, Razani, & Ponton, 2007; Carstairs, Myors, Shores, & Fogarty, 2006). Interpretation through (formal or informal) interpreters is often needed to assess these patients in their native language. Second, low education levels or illiteracy are common among (older) people in various minority ethnic groups in EU-15 countries (Blakemore et al., 2018; Danmarks Statistik, 2018; Schellingerhout, 2004). For example, more than 80% of Moroccan first-generation immigrants in the Netherlands did not complete any form of formal education (Schellingerhout, 2004). Illiteracy, a limited number of years of education, as well as a low quality of education significantly impact neuropsychological tests scores across several cognitive domains (Ardila et al., 2010; Ardila, Rosselli, & Rosas, 1989; Manly, Jacobs, Touradji, Small, & Stern, 2002; Nielsen & Jørgensen, 2013; Reis, Faisca, Ingvar, & Petersson, 2006; Reis, Petersson, Castro-Caldas, & Ingvar, 2001; Rosselli & Ardila, 2003; Rosselli, Ardila, & Rosas, 1990). Patients who are illiterate may also experience more discomfort in testing situations due to unfamiliarity with the setting, the content of the tests, or due to differences in what is considered a good response (Fujii, 2018). Third, neuropsychologists in EU-15 countries may encounter substantial cultural barriers in their clinical practice. In particular, the “guest workers,” who came to EU-15 countries as labor migrants in the post-World War II period, may have limited levels of acculturation to the dominant culture as they were initially expected to return to their countries of origin after a number of years—often resulting in a delay of decades in the development of policies promoting social integration and acculturation (Crul & Doomernik, 2003; Martin, 2014; van Mol & de Valk, 2016). Cultural differences may impact the neuropsychological assessment in several ways. The patient may have different expectations of (the purpose of) the assessment, of what is relevant information, and of what information may be shared with a stranger (Greenfield, 1997). Additionally, culture influences communication styles, idioms of distress, and the way symptoms may manifest themselves (Fujii, 2018). In addition, Al-Jawahiri & Nielsen (2020) showed that lower levels of acculturation are associated with poorer performance on tests of mental speed and executive functioning—even when tests were administered in the person’s native language and scores were corrected for other demographics. Furthermore, culture and acculturation may influence test scores when Western items are used in tests (e.g., in naming tests) or when the tests involve culture-specific testing elements and strategies (Ardila, 2005, 2007a, 2007b; Rosselli & Ardila, 2003).

In sum, language, (quality of) education, literacy, and culture substantially influence neuropsychological assessment. Thorough adaptations or newly developed neuropsychological tests are needed, but such tests are often lacking (Franzen et al., 2020). Although neuropsychologists in several of the countries of origin of minority ethnic patients are working on the validation of cognitive tests, these initiatives mostly seem to focus on tests originally designed for (educated) populations in North America and Europe, such as the trail making test (Oumellal, El Alaoui Faris, & Benabdeljlil, 2018) or the Montreal Cognitive Assessment (Azdad, Benabdeljlil, Al Zemmouri, & El Alaoui Faris, 2019) in Morocco, and tests from the BİLNOT battery in Turkey (Karakaş, Erdoğan Bakar, & Doğutepe Dinçer, 2013). Furthermore, people who are low educated or illiterate were not included in these validation studies or in the normative data samples.

Taking these barriers into consideration, administering a cross-cultural neuropsychological assessment requires neuropsychologists to acquire culture-competent skills and knowledge. Some general directions for training of psychologists are presented in the “Guidelines on Multicultural Education, Training, Research, Practice, and Organizational Change for Psychologists” by the American Psychological Association (2003). However, these guidelines are not specific to neuropsychologists. Additionally, EU- and USA-based neuropsychologists state that the ability to handle cultural diversity is a “vital functional competency” for clinical neuropsychologists worldwide (Hessen et al., 2018) and, more specifically, “one of the foundational entry-level competencies for neuropsychologists” (Smith, 2018). However, no details are provided on the specific knowledge or skills that EU-based neuropsychologists should acquire to attain sufficient competence to handle the substantial barriers in culture, language, and education.

All these factors pose challenges to the neuropsychological assessment of patients from minority ethnic groups. The question thus arises how neuropsychologists in EU-15 countries have adapted their clinical practice to the growing population of patients from minority ethnic groups. The first aim of this study was therefore to investigate the current state of the field of adult cross-cultural neuropsychological assessment in EU-15 countries. A second aim was to generate recommendations for researchers and policy makers on the main issues that should be addressed and the potential ways to approach these issues.

Methods

To systematically gather expert opinion data and reach a consensus among these experts, a Delphi study method was used, focusing on the former EU-15 countries—Belgium, Denmark, Germany, Finland, France, Greece, Ireland, Italy, Luxembourg, the Netherlands, Austria, Portugal, Spain, the UK, and Sweden. These EU-15 countries share a history of similar immigration patterns with prominent (de)colonialization (Engerman & Sokoloff, 2012; van Mol & de Valk, 2016) and post-World War II labor immigration (Diez Guardia & Pichelmann, 2006; van Mol & de Valk, 2016). The following definition of minority ethnic people was used in this study (Nielsen et al., 2011): “people who are first-generation immigrants or refugees from countries outside the extended EU, Canada, USA, Australia or New Zealand.” The study was split into three rounds (Hsu & Sandford, 2007). The first two rounds consisted of web-based surveys in which the panelists were blinded to other the panelists’ responses. The last round was a face-to-face meeting. As not all panelists could participate in the face-to-face meeting due to time and distance constraints, an additional video conference was organized.

Delphi Expert Panel Selection

Potential panelists from EU-15 countries were identified based on an extensive search of the international peer-reviewed literature about neuropsychological assessment in patients from minority ethnic groups over the past 5 years. Search terms in several languages were used to make sure all experts were identified. Panelists who published within the last 5 years were selected in order to ensure that the experts were still actively involved in their field. Included panelists were asked to identify any other relevant experts, a technique known as snowballing (Iqbal & Pipon-Young, 2009). Three panelists were included in the final panel based on this technique, two of which were additional experts from the same country who were asked to complement the expertise of the original panelist.

First Delphi Round: Determining Current Status and Collecting Qualitative Data

We drafted a survey containing seven sections aimed at exploring the current status of the field (see supplementary material). The first section inquired about general panelist data and data on the clinic in which they worked. The second section gathered information about the minority ethnic groups visiting the panelists’ clinics. The neuropsychological assessment of these patients was detailed in the third section. The use of interpreter services was the topic of the fourth section. The fifth section discussed training for neuropsychologists, specifically concerning cross-cultural aspects. The sixth section examined the assessment of the following nine cognitive domains in patients from minority ethnic groups: language, memory, working memory, visuospatial functioning, orientation (time/place), attention, mental speed, executive functioning, and social cognition—largely following Lezak et al.’s (2012) classifications of cognitive domains. The last section requested the panelists to provide their recommendations for researchers and policy makers. A pilot version of the survey was emailed to two neuropsychologists with ample experience in cross-cultural neuropsychology, after which minor adjustments were made. The final survey was distributed in May 2019.

Second Delphi Round: Rating and Ranking Priorities

After the data collection for the first survey was complete, the results were integrated into a presentation format and sent by email to all panelists. A second survey was then drafted based on the results of the first survey (see supplementary material). The main aim of the second survey round was to rate and rank various priorities identified in the first survey. The second survey contained four sections. The first section asked panelists to rate the importance of each of the clustered general recommendations generated in the first survey. The second section asked panelists to rate the importance of having an interpreter present for patients with little, some, and a good understanding of the test administrator’s language. Furthermore, panelists rated the importance of having a trained interpreter and the importance of having a formal interpreter—as opposed to an informal interpreter. In addition, this section contained an open-ended question on how to improve the use of interpreter services during a neuropsychological assessment. The third section required panelists to rate the importance of training programs to become a neuropsychologist and the importance of training in cross-cultural neuropsychological assessment specifically. It also contained an open-ended question about the skills and knowledge required to carry out cross-cultural neuropsychological assessments. In the fourth section, panelists rated the importance of more research into the nine different cognitive domains and were asked to provide more detailed recommendations for research on this topic. The survey was distributed in August 2019.

Third Delphi Round: Confirming Consensus and Generating Further Recommendations

The third Delphi round was aimed at confirming consensus and discussing final recommendations. This round was split into two parts. One in-person meeting was held with four panelists at the conference of the Federation of European Societies of Neuropsychology in Milan, Italy, on September 7, 2019. An additional video conference was held on October 4, 2019 with six panelists, one of whom also attended the in-person meeting. The results of the first two surveys were sent to all participants before the meeting so they could first independently consider their opinions and relevant comments. Both meetings started with a summary of the main results of the survey rounds. During the meeting, all panelists were given turns to voice their opinions and/or comments. Group discussion of divergent views was encouraged. Panelists were all asked specifically about their opinion on the items that showed consensus in the second survey. Subsequently, all ratings for which wide quartile ranges remained after the second survey round were discussed in depth. All panelists consented to the recording of the meetings.

Results

Delphi Expert Panel

The number of panelists that participated in each round is displayed in Fig. 1. We could not identify experts from Luxembourg, Sweden, Finland, Greece, Portugal, or Ireland. A total of 16 potential panelists from the remaining EU-15 countries were approached for participation in the survey; three declined participation and two did not respond. One of the invited panelists joined only in the last phase of the study. In the second survey stage, one additional panelist was included. Overall, 12 experts from nine countries contributed to the Delphi study: one from Denmark, Germany, Belgium, England, Italy, and Austria and two from the Netherlands, France, and Spain.

Fig. 1.

Selection of panelists and participation in each Delphi round.

Round One: Determining Current Status and Collecting Qualitative Data

For half of the panelists, 5–15% of the patients in their clinic were estimated to be from a minority ethnic group, followed by 15–25% (three out of ten panelists). Some minority ethnic groups were seen in clinics in multiple countries, such as patients from Turkey (five clinics in four countries) and North African minorities (five clinics in four countries), whereas other populations were only seen by panelists from one country, such as South Asian and Afro-Caribbean minorities in the UK, Surinamese and Dutch Antillean minorities in the Netherlands, and Latin American people in Spain. The education level of the patients from minority ethnic groups was generally experienced to be lower compared to that of patients from the majority culture (mostly primary school education or lower in eight out of ten clinics).

Two panelists reported that they did not use any cross-cultural neuropsychological tests or test batteries; the other eight panelists all made use of one or more cross-cultural tests. The European Cross-Cultural Neuropsychological Test Battery (Nielsen et al., 2018) or its subtests were used in four out of nine countries. The Rowland Universal Dementia Assessment Scale (Goudsmit et al., 2018; Nielsen et al., 2013; Storey, Rowland, Basic, Conforti, & Dickson, 2004) was used in four out of nine countries. The Cross-Cultural Dementia screening (Goudsmit et al., 2017) was used in two countries. Additionally, the use of some tests seemed to be country-specific. For instance, modified versions of the Mini Mental State Examination (Folstein, Folstein, & McHugh, 1975; Rait et al. 2000a; Rait et al. 2000b) were used in the UK; the modified Visual Association Test (Franzen et al., 2019), a literacy screener (unpublished), and the Stick Design Test (Baiyewu et al., 2005) in the Netherlands; the computerized EMBRACED battery (unpublished) in Spain; the Cross-Linguistic Naming Test (Ardila, 2007b), the WHO/UCLA adaptation of the Rey Auditory Verbal Learning Test (Maj et al., 1993), and the Multicultural Cognitive Examination (Nielsen et al., 2019a) in Denmark; and the TNI-93 (Maillet et al., 2016), TMA-93 (Maillet et al., 2017), TFA-93 (Narme et al., 2019), and Montreal Cognitive Assessment (Nasreddine et al., 2005) in France.

Three out of ten experts stated that no normative data was available for the tests they used with patients from minority ethnic groups. Five panelists had either normative data for some, but not all tests, or normative data for some, but not all minority ethnic populations. Two panelists noted having normative data for all tests they used. Regarding the person administering the assessment, nine of ten experts reported that, generally, this person was not of the same ethnic background as the patient. To communicate with patients, six panelists used professional interpreters, whereas four panelists did not use interpreters. In Austria and France, interpreters were provided through governmental funding; in Denmark and Belgium, there were different rules depending on the specific case and context; in the other six countries, there were no government-funded interpreter services.

The panelists also rated the degree to which they could assess nine cognitive domains in patients from minority ethnic groups. A 10-point Likert scale was used, ranging from “I cannot assess this cognitive domain at all” (one) to “I can validly and reliably assess this domain and have a sufficient number of tests” (ten). Medians, first quartiles (Q1) and third quartiles (Q3) were extracted. As displayed in Fig. 2A, the domains of social cognition and language were ranked as the most challenging to assess. Various panelists indicated that no tools were available to assess social cognition in patients from minority ethnic groups. Similarly, language was described as hard or even impossible to assess. In contrast, memory was ranked the easiest cognitive domain to assess in these patients.

Fig. 2.

(A) Ease of administration ratings from the first round (% of panelists; medians, first quartile [Q1] and third quartile [Q3]). (B) Importance ratings from the second round* (% of panelists, first quartile [Q1] and third quartile [Q3]).

Clinical training to specialize as a neuropsychologist was available in six out of nine countries. Four panelists described that training in cross-cultural neuropsychological assessment is a part of clinical training, but that it is limited, e.g., voluntary. The panelists from Spain and Denmark described how they, or their clinic, provided their own training on the topic. Only the panelists from France described that cross-cultural assessment was a mandatory part of training in university.

Round Two: Rating and Ranking Priorities

In round one, a list of general recommendations was generated (see Table 1). These recommendations were then grouped into five broad categories. The first category, “Clinic and staff,” contained recommendations such as the employment of ethnically diverse neuropsychologists. The second category, “Training, knowledge, and awareness among neuropsychologists” stressed the importance of cross-cultural knowledge and skills training for neuropsychologists. “Tests,” the third category, recommended more research into educational and cultural effects on test performance and for the development of new tests. “Norms,” the fourth category, recommended development of normative data that takes into account education, culture, and country of origin, and for tests for which normative data are not required. The last category, “Other,” contained a variety of recommendations to improve assessment of patients from minority ethnic groups, such as additional resources in terms of assessment time and interpreter services, and research into specific populations, such as patients with mild cognitive impairment. In the second survey, all panelists were asked to rate the importance of these categories. The category “Tests” was ranked as the most important priority (median: 10, Q1-Q3: 9–10), closely followed by “Norms” (median: 9, Q1-Q3 8–10), and “Training, awareness, and knowledge among neuropsychologists” (median: 9, Q1-Q3 8–10). The recommendations from the “Other” category were ranked as less important (median 7, Q1-Q3: 7–8). Ratings of the “Clinic and staff” category showed heterogeneous responses, indicating limited consensus (median: 6, Q1-Q3: 3–8).

Table 1.

General recommendations

Clinic and staff	Training, knowledge, and awareness among neuropsychologists	Tests	Norms	Other
Employment of neuropsychologists with a diverse ethnic background	Professional training in cross-cultural neuropsychology for neuropsychologists, perhaps as a fixed part of specialist training for clinical neuropsychologists, including training on how to work with interpreters	More research on the way education and culture influence neuropsychological test performance for different cognitive domains, so we are better able to adapt neuropsychological tests and/or develop new tests	Development of normative data, specifically those that take into account education, culture, and/or country of origin	More resources needed, such as interpreters that are trained to work with neuropsychologists, funded by the government; more time to carry out the assessment
Employment of native speakers of minority ethnic languages or neuropsychologists with proficiency in different languages	Increased awareness among neuropsychologists of the influence of factors such as diversity, education, and culture on clinical practice	Develop and/or validate (fine-grained) neuropsychological tests, especially for cognitive domains for which there are few tests	Development of tests for which culture- or education-specific normative data are not required	More research with regard to specific populations, such as rare diagnoses, mild cognitive impairment, oldest old patients, semi-literate/illiterate patients
Providing services in a specialized, expertise setting, e.g., memory clinic with focus on minorities	Increase in knowledge of cultural aspects			Examining ways to instruct/inform patients from minority ethnic groups about the neuropsychological assessment (test-wiseness)
				Using additional biomarkers in diagnostic assessment

After having been presented with the results of the first survey, the panelists were asked to rate the need of further development of tests within the nine cognitive domains. Figure 2B displays the results, plotted on a reverse axis to facilitate the comparison with Fig. 2A as the answers were formulated in reverse directions for survey 1 and 2—i.e., the degree to which domains could be assessed versus the need for more research. Except for memory, the quartile ranges for all cognitive domains were smaller in the second than the first round, indicating a shift towards consensus. After the second round, the priorities of the cognitive domains could be grouped in three levels. Social cognition and language were ranked as most important for further development. The second most important set of cognitive domains was executive functioning, visuospatial functioning, working memory, and orientation. The domains of attention, mental speed, and memory were indicated to need the least amount of research. Specific recommendations from round one and two for the development of cognitive tests are provided in Table 2. Furthermore, the panelists made recommendations regarding the specific knowledge and skills that neuropsychologist needs to perform cross-cultural assessment (Table 3).

Table 2.

Recommendations from all Delphi rounds for research in each cognitive domain

	Social cognition	Language	Executive functioning	Mental speed	Attention	Visuospatial	Memory	Working Memory	Orientation	Other
Suggestions regarding tests	Emotion recognition	Good naming test suitable for various minorities, with adequate normative data, brief, efficient, sufficiently sensitive and specific, with little need of explanations in administration	Tests of planning, mental flexibility, inhibition and other (sub)functions of executive functioning	Tests less influenced by education and literacy	Tests less influenced by education and literacy	Perhaps use a “format that everyone comes across in life, e.g., a broken cup?”	Tests for episodic memory			Intelligence tests measuring aspects similar to reasoning or matrices scales
	Test capturing social norms and moral/social dilemmas	A test for the evaluation of speech, such as a complex picture description	Tests less influenced by education and literacy			Tests less influenced by education and literacy	Verbal and nonverbal			Test for suboptimal performance (malingering)
	Theory of mind		More tests for very low educated populations			More alternatives	Normative data for other minorities, e.g., Eastern Europeans, and for higher educated patients			Test for motor coordination
Other comments	Needs more research in general	“Will be hard to assess”	Variation between cultures and between individuals	Cultural norms and expectations influence test results, i.e., speed is a cultural norm		“Lack of alternatives to current tests […] Not sure which ones are best for different ethnic groups.”		“There are various options for working memory tests, but they have not been validated yet”	Move beyond MMSE orientation questions
	“Seriously lacking”; “huge need to develop them”	“Some language barriers are unbridgeable for an assessor that does not speak the participant’s language”	Need to understand how education modulates executive functioning in healthy population before examining culture			“I think we need to develop tests other than drawings or figure copy that require past school learning of geometry”		“We need to come up with a new testing paradigm with better ecological validity that does not rely on numbers”
	Should (also) be studied with more qualitative methods	“Most apparent barrier”
	Highly dependent on culture

Table 3.

Recommendations for knowledge and skills required by neuropsychologists

Knowledge about patients from minority ethnic groups	Culture, education, language and the assessment	Availability of cross-cultural tests	During/after the assessment	Recognize limitations	Reflect on own culture	Training staff
Knowledge about the cultural/linguistic background of main minority groups; background knowledge about the country of origin, migration history, customs, expectations, social roles and rules, religion, and traditions; knowledge about differences between cultures	Knowledge about how language, culture, and education influence test performance; knowing how to interpret test performance, e.g., what is normal performance for someone without education	Knowledge of which cross-cultural tests and batteries are available, perhaps with the help of test publishers?	Get used to working with formal and informal interpreters; know how to instruct them.	Recognize the limits of instruments and normative data.	Reflect on own culture; be aware of own prejudices and cultural beliefs; be aware of how others may perceive you	Be trained as a neuropsychologist before practicing neuropsychology
Knowledge about (cultural) views on cognition and cognitive impairment	Be aware of the cultural values underlying cognitive assessment		Be flexible, patient, listen well and be creative to work around barriers	Recognize the limits of own expertise and knowledge	Have an open mind	Training of test administrators by neuropsychologists
	Be aware of the necessity of cross-cultural tests and normative data		Know how to handle cultural barriers
	Knowledge of potential biases of instruments and assessment procedures; knowing which tests are culture-specific		Being able to do a broad (history taking) interview and use questions to assess cognitive functioning in an ecologically valid way
	Knowledge about the influence of culture on the brain		Being able to qualitatively interpret results

Regarding interpreters, panelists indicated it was critical to have an interpreter present for the assessment of patients with little understanding of the test administrator’s language (median: 9, Q1-Q3: 7–10). There was little consensus about the use of interpreters for patients with some understanding of the language (median: 7, Q1-Q3: 5–9) and for patients with a good understanding of the language (median: 5, Q1-Q3: 2–7). Having a formal interpreter present, as opposed to an informal interpreter such as a relative, was rated as important (median: 8, Q1-Q3: 6–9). Furthermore, having an interpreter present who is trained at interpreting during a neuropsychological assessment was rated as important (median: 8, Q1-Q3: 7–9). In the open-ended questions of the second survey, a list of recommendations to improve assessment with interpreters was generated (Table 4).

Table 4.

Recommendations for working with interpreters

1. All neuropsychologists should be trained in working with interpreters to improve the outcome of interpreted assessments

2. Neuropsychologists need to know about existing guidelines for working with interpreters—perhaps these should be included in standard (clinical) training

3. Interpreters should be trained

4. Standard practice before the neuropsychological assessment should entail a briefing with the interpreter about:

- the aims of the assessment/what the neuropsychologist wants to accomplish

- the case

- the instruments

- the procedures (of the assessment), in particular:

○ failing of the patient is a vital part of the examination

○ interpreters should translate as literally as possible

○ only to intervene/correct if the neuropsychologists says so, i.e., not give any hints, additional information, etc.

- the expected responses

5. Hire interpreters with experience/training in clinical settings, perhaps even create a whitelist (or blacklist) of interpreters

6. Adapted tests are often translated/back-translated multiple times; have interpreters use the official translated instructions instead of interpreting freely based on what the neuropsychologist says

7. Awareness of regional or country-specific variations in language; e.g., it is not ideal to use a Spanish interpreter from Spain for the assessment of Latin American patients

8. Improve availability (funding)

Round Three: Confirming Consensus and Additional Recommendations

During the face-to-face meeting, the panelists first reached a consensus on the importance of improving or developing cross-cultural tests and normative data and on increasing cross-cultural knowledge and training among neuropsychologists. Subsequently, the lack of consensus about the category “Clinic and staff” was discussed. This lack of consensus was suggested to be due to the limited feasibility of providing same-ethnicity staff in clinics with patients from a wide variety of ethnic backgrounds.

Second, the panelists agreed on the order of importance of development of cross-cultural cognitive tests for the nine cognitive domains. According to the panelists, heterogeneity in ratings on memory tests was caused by differences in the availability of cross-cultural memory tests across EU-15 countries. The results from the second survey indicated that each cognitive domain was assigned either a high, medium or low(er) priority. According to one of the panelists, this three-level hierarchy may reflect differences in conceptual complexity of these cognitive domains: memory and processing speed may be less challenging to capture in a cross-cultural cognitive test than concepts like executive functioning or social cognition. However, the question was raised whether it is possible to develop tests of social cognition and language that are cross-culturally and cross-linguistically applicable, or whether tests should be developed for each individual minority ethnic group. The practicality of this last approach was judged to be limited. Although some large minority groups may be present across various EU-15 countries (e.g., the Turkish minority), the patient population in most clinics is too diverse to have any use for tests that are specific to any individual minority culture or language.

Third, as indicated by the surveys, the panelists agreed on the need for an interpreter for patients with little understanding of the host country language, but no consensus was reached about the need for an interpreter for patients with some or a good understanding of the host country language. One panelist suggested that this probably depends on the level of precision needed for an assessment—an interpreter will be necessary to identify mild cognitive deficits, but may not be necessary to identify more severe cognitive impairment. Although the panelists recognized that there are inherent challenges associated with doing neuropsychological assessments with an interpreter, such as the risk of biased test results, assessment with an interpreter was often necessary as the availability of (same-ethnicity) neuropsychologists fluent in the patient’s language was very limited in the panelists’ countries. Although one panelist mentioned that patients may feel more at ease with a relative doing the interpretation, the panelists agreed that, generally, the use of formal interpreters was preferable to the use of informal interpreters. The panelists voiced their concerns regarding the ethical aspects of the quality of the interpretation and the potential bias introduced when using informal interpreters. According to the panelists, when assessing a patient with a formal interpreter, neuropsychologists need to be aware of various potential barriers. First, patients may feel ashamed about the fact that they are low educated and speak local, rural dialects, rather than speaking the more formal language of the interpreter. A second potential barrier mentioned by the panelists is a mismatch between the gender of the interpreter and the patient. A third issue is the variable quality of formal interpreter services in some countries, where interpreters do not always have “proven efficacy” and may only work as an interpreter for a short time. This is in contrast with some other EU-15 countries, where formal interpreters have to meet various criteria and follow a clear code of conduct. Finally, the use of telephone interpreters was identified as a challenge. The panelists agreed that evaluation with a telephone interpreter should be avoided, mostly because the interpreter cannot see the test materials that are used in the assessment.

Fourth, the panelists reached a consensus that (cross-cultural) training was important. It was recommended that this training should include both theoretical and practical training in cross-cultural assessment and working with diverse patients. No consensus was reached about how this training was best implemented as there were significant cross-country variations in neuropsychological training, certification, and licensing in general. Furthermore, the way expertise in cross-cultural neuropsychology was organized varied by country: some countries had expert centers, e.g., a specific multicultural memory clinic, whereas others had more “local expertise.” Only in France was a more extensive cross-cultural training provided to neuropsychologists.

In addition to reaching a consensus on these topics, a number of other relevant aspects were mentioned regarding the assessment of minority ethnic populations. One panelist commented that neuropsychologists should be aware of the effect of examiner–examinee ethnic discordance, as well as mentioning the possible effects of stereotype threat on cognitive test performance. Another panelist mentioned the inter- and intra-individual variability in the proficiency in, and use of the majority and minority languages, and the prestige that can sometimes be attached to proficiency in certain languages. Furthermore, two panelists mentioned that people from a different culture will not be familiar with undergoing a neuropsychological assessment, possibly inducing shame or (di)stress in patients, or making the patient feel treated like a child. During the meeting in Milan with panelists from Denmark, Italy, the UK, and the Netherlands, the costs of and access to neuropsychological services were also discussed. The panelists indicated that the assessments were either free or were covered by (mandatory) health insurance. In some countries, the availability of specialized services for minority patients depends heavily on whether patients live in the catchment area. Last, two of the panelists mentioned that, in patients from minority populations, it is important to take a wide range of variables into account: culture, age, gender, education, and lifetime (socio)demographic characteristics.

Discussion

The aims of this Delphi study were to examine the current state of the field of adult cross-cultural neuropsychological assessment in EU-15 countries and to generate recommendations for researchers and policy makers. The results showed that a number of instruments and batteries are available in EU-15 countries—in particular cross-cultural memory tests—several of which are currently used in more than one country. A consensus was reached that training of neuropsychologists and the development of cross-cultural tests and normative data are the most pressing matters. A consensus was reached on social cognition and language tests as the first priorities, followed by tests of executive functioning, visuospatial functioning, working memory, and orientation. The panelists agreed that tests that can be used across a variety of minority ethnic groups are preferable over tests specific to one culture or language. The panelists recognized that the use of formal interpreters is important, although neuropsychological assessment with interpreters may never be free of bias. Various recommendations were provided for working with interpreters and for training in cross-cultural neuropsychological assessment.

This study showed that considerable work has already been carried out in the development and validation of cross-cultural neuropsychological tests in Europe. In particular, the European CNTB and the RUDAS are well validated; these instruments have been studied in people from numerous minority groups, with a wide variety of education levels, in studies from across multiple European countries (CNTB: Al-Jawahiri & Nielsen, 2020; Nielsen, 2019; Nielsen et al., 2018; RUDAS: Goudsmit et al., 2018; Nielsen et al., 2013; Nielsen et al., 2019b). Together, these instruments measure a variety of cognitive functions: general cognitive functioning (RUDAS), memory (Enhanced Cued Recall and Recall of Pictures Test), language (Picture naming and semantic verbal fluency), executive functions (Color Trails Test and Five Digit Test), and visuospatial functions (Clock Reading Test, Clock Drawing Test and copying of simple and complex figures). For some of the other instruments identified in this study, few (if any) validation studies in the target population have been published. Most experts reported using one or more cross-cultural (adapted) cognitive test, but few panelists were familiar with all the tests that were used by the other experts. This highlights that existing tests validated in one country should be better publicized, reviewed, and implemented in other EU countries. This will, at a minimum, require carrying out local validation studies following international standards, such as those of the International Test Commission (International Test Commission, 2017), as well as negotiating with publishers. The development of new cross-cultural tests and normative data was rated as highly important. Merely stratifying normative data by age and education may be insufficient for low educated patients from minority ethnic groups, who often show floor effects on neuropsychological tests requiring any form of school-based procedures (Nielsen, 2019). The development of new neuropsychological tests is therefore warranted. To suit a diverse patient population, the international literature recommends designing tests without black-and-white line drawings (Franzen et al. 2019; Reis, Faisca, Ingvar, & Petersson, 2006; Reis, Petersson, Castro-Caldas, & Ingvar, 2001), culture-specific stimuli (Ardila, 2005, 2007a, 2007b), or test elements that require skills learned in school (Nielsen & Jørgensen, 2013). Additionally, tests that are more ecologically valid may be more suitable for this population (Ardila, 2007a; Franzen et al. 2020). As developing tests and collecting normative data can be a costly and time-consuming process, researchers will have to prioritize which cognitive domains to tackle first. The experts in this study particularly agreed on a general lack of tests measuring social cognition and language, as opposed to some of the other cognitive domains for which more tests are available. Memory was the domain that panelists considered to be lacking the least in terms of test development. This finding reflects the better availability of cross-culturally validated memory tests in EU-15 countries, which is probably due to the relative ease with which memory tests can be developed or adapted to suit minority ethnic groups, e.g., by using items that are common and familiar to the minority ethnic group and by presenting them in a suitable format (Lim et al., 2009). Adequately validated tests and normative data for the cognitive domain of social cognition generally seem to be lacking in the EU-15, even for native-born adults (Buhl, Stokholm, & Gade, 2013; Merceron & Prouteau, 2013). Aside from social cognition and language, one panelist also suggested to validate or develop performance validity tests. A similar call to action was made at the Sixth European Conference on Symptom Validity Assessment in 2019, stressing that the cross-cultural validity of current performance validity tests is probably limited (Merten, 2019). Panelists from two countries in this study were working on, or had previously worked on, cross-cultural validation studies of performance validity tests (Daugherty et al., 2019). However, no true experts seem to exist in the EU that specialize specifically in the topic of performance validity testing in minority ethnic populations in Europe—research on this topic currently seems to be dominated by work from other regions (Nijdam-Jones, Rivera, Rosenfeld, & Arango-Lasprilla, 2017, 2019; Nijdam-Jones & Rosenfeld, 2017).

Aside from looking into cognitive tests themselves, it is important to take the cultural context of neuropsychological assessment into consideration. These contextual factors are elegantly summarized by the acronym of the ECLECTIC framework (Fujii, 2018): Education and literacy, Culture and acculturation, Language, Economics (e.g., socioeconomic status), Communication, Testing situation, comfort and motivation, Intelligence conceptualization, and Context of immigration. Although the (design of the) current study mainly highlighted the importance of cognitive tests and norms, several key contextual factors were mentioned by the panelists. For example, an unpublished literacy screening test was used in one country to determine the quality of the patients’ education (E). Neuropsychologists from a number of European countries make use of short acculturation scales (C) in their clinics—such as a modified version of the Short Acculturation Scale for Hispanics (SASH; Marin, Sabogal, Marin, Otero-Sabogal, & Perez-Stable, 1987). Additionally, the effects of language abilities in both native and host country languages (L) was mentioned, as well as the effects of stereotype threat (Steele, 1997), of being unfamiliar with cognitive testing, and of examinee-examiner ethnic discordance (T) on the assessment. The panelists also mentioned it is important to take into account lifetime (socio)demographic factors and access to and availability of health services (E). Some aspects from the ECLECTIC framework, in particular communication styles and intelligence conceptualization, received less explicit attention in this study. This may in part be due to the way the surveys were designed (i.e., with a relatively heavy emphasis in the forced-choice questions on cognitive tests). Other specific examples of relevant issues to take into consideration in working with minority ethnic groups are traumatic experiences and migration-related distress or grief (Carta, Bernal, Hardoy, Haro-Abad, & the “Report on the Mental Health in Europe” Working Group, 2005), differences in explanatory models of illness (Fazil, Wallace, & Hussain, 2006; van Wezel et al., 2018), exposure to discrimination (de Freitas, Fernandes-Jesus, Ferreira, & Coimbra, 2018), and differences in symptom manifestation and idioms of distress, such as mixed affective and somatic presentations of depression in Moroccan and Turkish patients (Sempértegui, Knipscheer, Baliatsas, & Bekker, 2019).

Cross-cultural neuropsychological assessment could benefit from matching patients from minority ethnic groups with same-ethnicity neuropsychologists. The experts in this study agreed that providing same-ethnicity neuropsychologists to all patients from minority ethnic groups in the EU-15 countries is currently not feasible considering the number of different minority ethnic groups and the limited ethnic diversity among neuropsychologist in the EU, which is in line with the reality in the USA (Elbulok-Charcape, Rabin, Spadaccini, & Barr, 2014). Instead, the panelists identified more cross-cultural training, awareness, and knowledge among neuropsychologists as an important need for cross-cultural assessment. Cross-cultural training of neuropsychologists was also identified as a priority in the USA, where “clinicians often lack in-depth training in assessment of ethnic minorities” (Mindt, Byrd, Saez, & Manly, 2010). In the present study, a list of important knowledge and skills was generated for training in cross-cultural assessment. These recommendations can be supplemented with existing guidelines, such as those captured in the ECLECTIC framework (Fujii, 2018). With regard to the implementation of these recommendations, a recent study indicated that training to become a neuropsychologist is organized differently across the EU, and the duration of training varies substantially between 12 and 60 months (Hokkanen et al., 2019). The way cross-cultural skills are incorporated in neuropsychology training may thus have to be decided separately for each country. Alternatively, cross-cultural training could be realized by organizing a European summer school in cross-cultural neuropsychology, e.g., in collaboration with the Federation of European Societies of Neuropsychology (FESN) or the European Federation of Psychologists’ Associations (EFPA).

Concerning the use of interpreters, the panelists generally agreed that having a formal interpreter present—as opposed to an informal interpreter, such as a family member—was important. However, the panelists also agreed that various challenges will remain, even with a formal interpreter present. Previous studies have similarly indicated that working with interpreters carries risks. The use of relatives as interpreters has been related to the exclusion of the patient from the conversation (Zendedel, Schouten, van Weert, & van den Putte, 2018), problems with the adequate translation of medical terminology (Manly & Espino, 2004), obscuring of the patient’s explanatory models, and difficulties in assessing the level of insight (Kilian, Swartz, Dowling, Dlali, & Chiliza, 2014). The use of formal interpreters may be challenging as well, especially for tests with high demands on the abilities of the interpreter or when interpreters have received little formal training (Casas et al., 2012). The use of telephone interpreter was discouraged by the panelists as the interpreters would be unable to see the test materials. Additionally, assessment with a telephone interpreter can be hindered by factors such as disturbances in communication due to background noise (Hadziabdic, Heikkila, Albin, & Hjelm, 2011). We believe that EU guidelines for working with interpreters in the neuropsychological assessment of patients from minority ethnic groups are needed and that these could be extensions of existing guidelines, such as those of the British Psychological Society (Tribe & Thompson, 2017).

Some limitations to this study should be acknowledged. First, the total number of experts that could be identified (12) was relatively small—a typical Delphi study will have between 10 and 50 panelists (Turoff, 2002)—and a total of six EU-15 countries were not represented in the panel—Finland, Greece, Ireland, Luxembourg, Portugal, and Sweden. This finding seems to indicate that the field of cross-cultural neuropsychological assessment is largely still a developing field in the EU-15, and formal expertise is localized, rather than widespread. Additionally, the panelists identified using the criteria in the search strategy were nearly invariably of a majority background. Additional research is needed to determine whether the main findings of this study are endorsed by neuropsychologists with a minority ethnic background. This might be accomplished by broadening the inclusion criteria in a follow-up study, such as by replacing the publication criterion with other indicators of expertise, e.g., by peer-nomination, self-report, or based on having assessed a specific number of minority ethnic patients. Second, it would have been preferable if all panelists could have participated in one final face-to-face round, which was not possible due to time and distance constraints. By splitting the third round in two meetings, a risk of bias may have been introduced, as smaller groups tend to be more vulnerable to individual panelists holding strong opinions. However, we estimate that these effects were probably minimal, given that (a) all panelists received the survey results before the meeting, so they could independently form their opinions, (b) all panelists were given turns to speak, (c) group discussion of divergent views was encouraged during the meetings, and (d) panelists did not reach a consensus on all topics, indicating group pressure to conform was probably negligible. Another limitation of the study was that the majority of the experts worked in a memory clinic setting—although several of them also had experience with assessment of either healthy people from minority ethnic groups or patients from minority ethnic groups in other settings than memory clinics. The overrepresentation of memory clinic experts may partly be influenced by the snowballing technique used in the study, but could also reflect the predominant focus on dementia research in the EU, possibly due to dementia’s large economic and societal costs (Takizawa, Thompson, van Walsem, Faure, & Maier, 2015). A last limitation of this study is that no specific metric intervals were determined to define consensus at the outset of the study.

Given the aforementioned limitations, the results from this study should be seen as a first step towards the development of new policies. More research is needed to ensure that minority groups are represented and their opinions heard. We suggest broadening the scope of this study to represent more neuropsychologists with a minority background, as well as non-expert neuropsychologists, cultural psychologists, and community stakeholders to bring to light all relevant needs and perspectives. Furthermore, the population of interest should be expanded to include immigrants in the wider EU, transnational European minorities, such as Roma people across Europe, and second and third-generation descendants of immigrants. The second generation is more often bilingual and higher educated than the first generation of immigrants, although notable heterogeneity within this group exists—for example, second-generation Turks more often lag behind on Dutch language fluency and are often lower educated than their Moroccans peers in the Netherlands (Crul & Vermeulen, 2003). It will be a challenge to determine which tests and normative data will be most appropriate for this heterogeneous population.

In conclusion, this study indicates that significant work has been carried out in the development and validation of cross-cultural neuropsychological tests in Europe. However, despite recent advances in cross-cultural neuropsychological testing and training in some EU-15 countries, this Delphi expert study highlights the continuing need for development of cross-cultural tests and normative data as well as culture-sensitive training, awareness and knowledge among European neuropsychologists. To improve the field of cross-cultural neuropsychology across the EU-15, countries should increase collaboration—both within the EU and with neuropsychologists from the countries of origin of minority ethnic patients—to (a) exchange ideas and methods for cross-cultural neuropsychological assessment, (b) validate tests and collect normative data, and (c) collaborate in training approaches and the development of guidelines for working with interpreters.

Funding

This work was supported by The Netherlands Organisation for Health Research and Development (ZonMw Memorabel) [grant number 733050834 to SF, JMP, EvdB].

Conflicts of Interest

The authors report no conflict of interest.