Abstract
Objective
The objective of this study is to propose a TeleNP model for remote assessment and offer practical recommendations for clinical practice with patients in Mexico and Latin America, based on a systematic literature review and clinical experience.
Method
A systematic review of studies from 2011 to 2021 in English and Spanish used TeleNP, teleneuropsychology, telepsychology, online, assessment, teleneuropsicología, and evaluación for the search; the databases examined included PubMed, BiDi UNAM, ScienceDirect, Google Scholar, and Wiley One Library; the Oxford Centre for Evidence-Based Medicine system was used to grade the levels of evidence. The experience of the last two years of students and faculty in the Master’s and Doctoral Programs in Psychology, Clinical Neuropsychology Residency Program, was also used as a basis for this guide.
Results
We propose a clinical model for TeleNP assessment in Mexico and Latin America based on the review of 31 articles and the practice of professors and students of clinical neuropsychology.
Conclusion
The proposed model describes a procedure and adaptations for home-to-home clinical practice in the neuropsychological assessment of Mexican patients that could also be used in other Latin American countries. Its reliability remains to be assessed, but this model and the suggestions proposed could be used in future studies and clinical trials for Mexican and Latin American populations.
Keywords: Teleneuropsychology, Neuropsychological assessment, Latin America, Telemedicine, COVID-19
Introduction
Direct contact with patients is fundamental to clinical practice in neuropsychological assessment (Koterba et al., 2020). However, limited mobility, difficulty with transportation, contagious disease, and distance from medical centers can be obstacles to face-to-face examination. Recently, such obstacles also include the distancing measures required by the SARS-CoV-2 virus that has been responsible for the worldwide COVID-19 pandemic. In many countries, a state of emergency was declared in 2020, with restrictions on nonessential activities that affected neuropsychological evaluation in hospitals, clinics, and educational centers for the practice of this specialty (Wells, Dukarm, & Mills, 2021). Practitioners have thus sought ways to carry out assessment and diagnosis remotely. Although Brearly et al. (2017), in a systematic review of the use of videoconferencing in neuropsychology, called attention to the advantages of this technology in evaluation, it has not been considered routine practice until more recently.
Teleneuropsychology (TeleNP) refers to the remote application of clinical neuropsychology practice using audiovisual technology for evaluation, diagnosis, and intervention (Bilder et al., 2020). TeleNP assessment can be carried out according to one of three models: clinic-to-clinic, where the practitioner and patient are in the same or different clinic but in separate rooms; clinic-to-home, where the practitioner is in the clinic and the patient is at home; and home-to-home, where the practitioner and the patient are in their respective homes (Fox-Fuller et al., 2021). The background to these methods is the suggestions and procedures for the practice of telepsychology of the American Psychological Association (APA; American Psychological Association, 2013) and of Grosch, Gottlieb, and Cullum (2011), who provide specifications for its use, criteria, special considerations, safety measures, and ethical considerations. There are also studies that describe the advantages and limitations of the approach (Sherwood & MacDonald, 2020; Stead & Vinson, 2019) and that evaluate its use with older adults (Brearly et al., 2017; Cullum, Hynan, Grosch, Parikh, & Weiner, 2014; Kirkwood, Peck, & Bennie, 2000; Stead & Vinson, 2019). However, it was not until the COVID-19 pandemic that organizations such as the International Neuropsychological Society (INS; Cullum, Hammers, Harder, & Stolwyk, 2020) and the International Organization Practice Committee (IOPC; Arias et al., 2020; Bilder et al., 2020; Inter Organizational Practice Committee, 2020; Postal et al., 2021) began to publish guidelines for remote neuropsychological assessment, or TeleNP.
Various systematic reviews (Chapman, Gardner, Ponsford, Cadilhac, & Stolwyk, 2021; Crivelli et al., 2021; Nussbaum, Young, DeLeon, Engelmann, & Schraegle, 2021; Palmese et al., 2022; Pérez, Ramos Usuga, & Arango Lasprilla, 2021) have shown a high correlation between the results of TeleNP and face-to-face assessment, but none of these have examined the Mexican or Latin American population. Some universities in the United States, including the University of California, Los Angeles, the Yale School of Medicine, the University of Utah, and Baylor College of Medicine, have implemented different models for assessing the Latino population in that country (Arias et al., 2020). These models have in common the use of informed consent and the exploration of technological criteria, as well as aspects of specific neuropsychological conditions to evaluate the reliability of TeleNP assessment. Although TeleNP has acquired major importance in addressing the restrictions regarding personal contact and social distancing during the COVID-19 pandemic, its use might be extended beyond that of immediate necessity: models for the implementation of TeleNP might also allow its use with people who need neuropsychological attention but who, because of distance or other circumstances, do not have access to this specialty.
Some of these suggestions and recommendations are difficult to apply in Mexico and other Latin American countries, for technological, social, and economic reasons (Pierce, Schroeder, & Suchecki, 2021). According to this, the World Bank and CEPAL (Comisión Económica para América Latina y el Caribe, CEPAL) report comparable rates of urbanization in Mexico and the rest of Latin America (Banco Mundial, 2021). The Organization for Economic Cooperation and Development (Organización para la Cooperación y el Desarrollo Económico, OCDE) estimated in 2020 that 21% of the urban population in Latin America and the Caribbean lived in marginal neighborhoods, informal settlements, or inadequate housing without the availability of basic services. However, CEPAL has reported an increase in mobile telephone ownership in the region with respect to the previous decade, up to 62% in some countries. In contrast, the increase in computer use was more modest (OECD & World Bank, 2020; Quiroga-Parra, Torrent-Sellens, & Murcia, 2017). Kohn et al. (2018) consider the mental health problems in Mexico to be comparable to those in Latin America in general; they highlight a mental health care infrastructure lacking in human and material resources, particularly mental health care providers and training programs. A TeleNP model adapted for Mexico and other Latin American countries could thus be useful in providing neuropsychology services to places where access is complicated and health personnel are limited.
The objective of this study is to propose a procedure model and guidelines for the use of TeleNP in assessing pediatric and adult patients in Mexico, based on scientific evidence from a systematic review of the literature for TeleNP assessment and the clinical experience of the authors. Such a model could also be used in other Latin American countries with similar conditions.
Method
Literature Search and Selection
To validate some of the procedures in the model, a systematic literature review was carried out based on the PRISMA 2020 model (Page et al., 2021; Checklist S1), using PubMed, UNAM Digital Library (BiDi UNAM), ScienceDirect, Wiley Online Library databases, and Google Scholar. The search terms used were TeleNP, teleneuropsychology, telepsychology, online, assessment, teleneuropsicología, and evaluación. The search included articles published from 2011 to 2021 in English and Spanish. It was carried out on December 17, 2021, and found a total of 159 abstracts: 156 in English and 3 in Spanish. Seventy-five were excluded as duplicates, leaving a total of 84. Articles were then selected if they (1) were group studies that evaluated TeleNP or (2) guidelines and recommendations for the TeleNP assessment, (3) involved any remote communication device (cell phone, videoconference, or others), and (4) described assessments carried out with either adults or children. Articles were excluded if they were reviews or meta-analyses, book chapters, conference proceedings or posters, case studies, group studies using only surveys or interviews, and studies whose objective was to validate a technological platform. The titles and abstracts were recorded in an Excel table, and two neuropsychology experts classified them as appropriate, not appropriate, or possible based on the inclusion and exclusion criteria. A Cohen’s k analysis was carried out and found a moderate level of agreement (k = .521, p < .001). Based on their classifications, 37 articles were excluded, and 23 were chosen by mutual agreement. The 24 on which the judges did not agree were examined by an expert who reviewed the full text of the papers. Eight of these were selected based on the objectives of this study, for a total of 31 selected articles (Fig. 1).
Clinical Experience during the COVID-19 Pandemic
The elaboration of this model and accompanying guidelines was based on the experience of professors and students during supervised clinical practice in the Master’s and Doctoral Programs in Psychology of the Faculty of Psychology of the Universidad Nacional Autónoma de México (UNAM), Clinical Neuropsychology Residency Program, Facultad de Estudios Superiores Iztacala (FES-I) campus, from August 2020 to December 2021. During this period, all neuropsychological assessments performed by faculty and students were carried out remotely, given the restrictions imposed during the COVID-19 pandemic, which included social distancing and the suspension of face-to-face activities in the program’s various locations for clinical practice.
Results of the Literature Review
The literature review included 13 cohort studies, 13 sets of guidelines, 3 sets of guidelines with literature recommendations, and 2 descriptive studies; 17 were carried out with adults, 7 with children, and 7 did not specify the age of the studied population; 30 were in English and only one in Spanish. As in Crivelli et al. (2021), the level of evidence was rated according to the classification system of the Oxford Centre for Evidence-Based Medicine (2011). The level of evidence of each article and a summary of the information obtained are shown in Table 1. In general, cohort studies report that TeleNP shows appropriate reliability and validity and that it can be used as an option for the evaluation of pediatric and adult populations, specifically to determine cognitive dysfunction. The guidelines refer to the use of informed consent, features and requirements of the online platforms utilized, procedures before, during, and after evaluation sessions, and the structure and delivery of results.
TABLE 1.
Author | Year | Level of evidence | Population | n | Age, mean (SD) |
Type | Objective | Platform | Results | Recommendations for TeleNP assessment |
---|---|---|---|---|---|---|---|---|---|---|
Alegret et al. | 2021 | 3a | Adults | 338 | 74 (9.3) |
Cohort study | NBACE reliability and sensitivity | Skype, FaceTime, Google Duo, and WhatsApp | Adequate feasibility and sensitivity for NBACE TeleNP No differences between platforms |
All platforms examined are useful for TeleNP |
Arias et al. | 2020 | 3a | Adults | Outcome research | Offer clinical guidance for TeleNP with U.S. Latino population | HIPAA-compliant platforms | Offers a TeleNP workflow | Model process for TeleNP (screening and assessment) | ||
Bilder et al. | 2020 | 5 | Both | Guide | Provide a rapid guide for TeleNP | Zoom | Offers a complete guide for TeleNP | A method for TeleNP assessment | ||
Chapman et al. | 2021 | 3a | Adults | 48 | 64.6 (10.1) |
Cohort study | Reliability of TeleNP for stroke survivors | Zoom | No differences between face-to-face and TeleNP assessment | Initial training, two cameras for patient, NP test modifications |
Crivelli et al. | 2021 | 2a | Adults | 19* | Guide and literature recommendations | Recommendations for TeleNP process in Latin America |
N/M | A guide for TeleNP process in a Latin American population based on SR |
Target population and TeleNP assessment process recommendations | |
Cullum et al. | 2014 | 3a | Adults | 202 | 68.5 (9.5) |
Cohort study | TeleNP battery feasibility for dementia |
ITU-T H.323 packet-based | VTC-based neuropsychological testing is valid and reliable for assessment |
Scoring of drawing tests in real-time |
Fox-Fuller et al. | 2021 | 3a | Adults | 44 | 33.98 (15) |
Cohort study | Home-to-home TeleNP feasibility | Zoom | Adequate feasibility for some tests in Home-to-home modality | Home-to-home TeleNP could be adequate for assessment |
Franco-Martin et al. | 2012 | 3a | Adults | 30 | 50.03 (5.89) | Cohort study | Clinic-clinic Spanish TeleNP feasibility | psicoED | Adequate TeleNP feasibility for psychotic patients | Motor adaptions for visuomotor assessment and recommendations for greater bandwidth |
Galusha-Glasscock et al. | 2016 | 3a | Adults | 18 | 69.67 (7.76) |
Cohort study | TeleNP battery feasibility | Polycom iPower | Adequate feasibility and reliability | 26-inch monitor and an assistant for assessment |
Gnassounou et al. | 2021 | 3a | Adults | 68 | 71.5 (5.9) |
Cohort study | TeleNP battery feasibility and satisfaction | MacBook Pro system |
Adequate satisfaction (87%) and feasibility | 2 cameras for assessment, 14-inch monitor, a trained assistant, and recommendation regarding TeleNP incidents |
Grosch et al. | 2011 | 5 | Both | Guide | Elaborate a guide to practical and ethical considerations for TeleNP assessment | Any system with HIPAA security rule | Ethical and TeleNP recommendations | Ethical suggestions for TeleNP assessment | ||
Harder et al. | 2020 | 3a | Pediatric | 58 | 13.10 (3.56) |
Cohort study | Evaluate home-based feasibility and satisfaction with TeleNP in pediatric population | VSee | Adequate feasibility and satisfaction | Recommendations for TeleNP in pediatric populations |
Hewitt et al. | 2020 | 5 | Both | Guidance | Elaborate a guide to transitioning to TeleNP | Zoom and Vidyo | Guide to transition to TeleNP NP assessment | Considerations for adult, pediatric, and forensic populations, and rehabilitation | ||
Kitaigorodsky et al. | 2021 | 5 | Adults | Guidance | Propose a TeleNP protocol to facilitate neuropsychological assessment utilizing a virtual platform |
Zoom | TeleNP model for adult assessment | Recommendations for TeleNP in adult population |
(continue)
TABLE 1.
Author | Year | Level of evidence | Population | n | Age, mean (SD) |
Type | Objective | Platform | Results | Recommendations for TeleNP assessment |
---|---|---|---|---|---|---|---|---|---|---|
Loman et al. | 2020 | 5 | Pediatric | Guidance | Provide pediatric neuropsychology processes and procedures for neuropsychology services during the COVID-19 pandemic | Teams, Zoom | Recommendations for TeleNP and face-to-face NP assessment during COVID-19 pandemic |
Recommendations for TeleNP in pediatric population | ||
Mahon et al. | 2021 | 3a | Adults | 30 | 23 (N/M) |
Cohort study | Examine the feasibility and reliability of home-based videoconference administration of the Wechsler Adult Intelligence Scale-4th Edition (WAIS-IV). |
Zoom | Adequate feasibility and reliability | Adjustments for WAIS-IV assessment |
Nguyen et al. | 2021 | 5 | Adults | Guidance | Describe the challenges related to TeleNP practices with Asian-American patients and offer practical recommendations to complement existing TeleNP guidelines | HIPAA-compliant platforms | Recommendations for TeleNP assessment in Asian-American populations |
Recommendations for TeleNP in Asian-American populations | ||
Nussbaum et al. | 2021 | 2a | Pediatric | Guide and literature recommendations | Review an assessment paradigm for conducting virtual neuropsychological pre-surgical epilepsy evaluations in the context of the COVID-19 pandemic | Zoom | Proposes models for presurgical epilepsy TeleNP assessment with possible reduction of infection | Three options for TeleNP assessment in pediatric epilepsy population | ||
Palmese et al. | 2021 | 2a | Adults | Guide and literature recommendations | Offer care management guidelines for TeleNP and face-to-face evaluation of pre-deep brain stimulation populations | Zoom, VSee, WebEx, StarLeaf, Caregility, and BlueJeans | Proposes three models for pre-DBS TeleNP assessment | Recommendations for home-based, clinic-based, and hybrid TeleNP | ||
Parks et al. | 2021 | 3a | Adults | 111 | 58.91 (14.30) |
Cohort study | Explore the validity and clinical utility of in-home TeleNP testing in a mixed clinical sample during the COVID-19 pandemic | Zoom | TeleNP test scores did not significantly differ from face-to-face testing | Feasibility of home-based TeleNP in adults and patients with mild cognitive impairment |
Parlar et al. | 2020 | 5 | Adults | Guidance | Propose a “contactless” evidence-based inpatient test battery to be used for in-person assessments amenable to physical distancing | N/M | Suggests a TeleNP battery and workflow for contactless inpatient neuropsychological evaluations |
Recommendations for inpatient assessment | ||
Peterson et al. | 2021 | 3a | Pediatric | Outcome research | Propose a novel three-tiered telehealth service model | Zoom | A guide for pediatric neuropsychology providers considering TeleNP services for complex pediatric patients during social distancing. | A three-tiered model for hybrid and TeleNP assessment | ||
Postal et al. | 2021 | 5 | Both | Guidance | Offer guidance on models to provide neuropsychological care during the COVID-19 pandemic | N/M | Recommendations and guidance with risk–benefit analysis of various models of neuropsychological care during the COVID-19 pandemic | Risks and benefits of hybrid and TeleNP | ||
Pulsifer et al. | 2021 | 3a | Both | Adults 141 Pediatric 25 |
Adults 58.35 (18.34) Pediatric 12.28 (4.90) |
Cohort study | Examine practice patterns within a lifespan outpatient neuropsychology center before, during, and after pandemic | Zoom | Some variables can affect TeleNP | TeleNP is an appropriate option for population with neurodegenerative conditions |
(continue)
TABLE 1.
Author | Year | Level of evidence | Population | n | Age, mean (SD) |
Type | Objective | Platform | Results | Recommendations for TeleNP assessment |
---|---|---|---|---|---|---|---|---|---|---|
Ransom et al. | 2020 | 3a | Pediatric | 129 | 8.78 (6.53) | Cohort study | Demonstrate the feasibility of TeleNP in a pediatric clinical population and disseminate clinical decision-making procedures to guide best practices for pediatric TeleNP | Teladoc Health system | TeleNP is feasible in a pediatric clinical population to begin the process of differential diagnosis and treatment planning |
Recommendations for TeleNP in pediatric population |
Scott et al. | 2021 | 5 | Both | Guidance | Provide ethical considerations and suggestions for potential practitioners of TeleNP based on new models of practice responding to the COVID-19 pandemic | HIPAA-complaint platforms | Model for deciding the type of NP assessment (clinic, home, or hybrid) based on ethical norms |
Comparison of three models of NP assessment and ethical recommendations for TeleNP | ||
Sherwood et al. | 2020 | 5 | Pediatric | Guidance | Expand clinical access to specialized pediatric services in rural areas | Zoom | Proposes a clinical service decision tree with specific criteria to guide providers on triage, to increase access to specialized healthcare | Recommendations of criteria for TeleNP candidates | ||
Singh & Germine. | 2021 | 5 | Both | Guidance | Introduce the term Hybrid Neuropsychology | N/M | Proposes a hybrid neuropsychology process for clinical practice | Considerations for assessment with a hybrid model |
||
Tailby et al. | 2020 | 5 | Adults | Guidance | Describe how the pilot study of the Australian Epilepsy Project (AEP) has transitioned to TeleNP, enabling safe operation during the pandemic |
Zoom | TeleNP enables large-scale neuropsychological research during periods of social distancing (and in other situations) and offers an opportunity to expand the reach and breadth of neuropsychological services | Adequate feasibility and recommendations for TeleNP assessment process for epilepsy population | ||
Thibodaux et al. | 2021 | 5 | Pediatric | Guidance | Present practical professional and ethical TeleNP assessments for pediatric population | Zoom | A model for pediatric TeleNP population | A four-step model (intake, connectivity visit, evaluation, and feedback) for pediatric TeleNP assessment | ||
Wadsworth et al. | 2018 | 3a | Adults | 78 | 72.71 (8.43) | Cohort study | Determine the ability of TeleNP to distinguish between cognitively impaired and unimpaired older adults | Polycom iPower 680 | TeleNP assessment is equally effective as face-to-face assessment in distinguishing between cognitively impaired and unimpaired individuals | Validate the TeleNP assessment for distinguishing neurocognitive disorders |
Note: NBACE = Neuropsychological Battery of Fundación ACE; TeleNP = Teleneuropsychology: N/M = Not mentioned.
* Articles
Most of the articles reviewed that assessed the reliability and validity of remote assessment and intervention examined clinic-to-clinic TeleNP; a minority analyzed clinic-to-home models, and only one analyzed the home-to-home model. It might be expected that the recommendations in guidelines for TeleNP could be applied to clinic-to-clinic or clinic-to-home models, but some of these recommendations could not be applied to the home-to-home model because of factors that are difficult to control in the home of the evaluator as well as that of the patient.
The findings in the literature related to TeleNP procedures are generally based on clinic-to-clinic models, use of informed consent, addressing patient or caregiver questions, telephone screening and follow-up, computer application of TeleNP, norms for the use of digital platforms, control of technical variables such as bandwidth, and the development of assessment platforms. In our practice, suggestions such as not using smartphones, having a place free of distractions, having adequate bandwidth and video quality, and some adaptations of neuropsychological test administration have been difficult to implement because of our population’s limited access to technological resources. A procedure was thus sought that included appropriate strategies for developing countries to make assessment simpler and more reliable. The model described below is a proposal for procedures applicable to home-to-home TeleNP, but we do not rule out its use for other modalities. It is based on the recommendations, limitations, and findings of the literature review, as well as the clinical expertise gained during the COVID-19 pandemic.
Proposal for a TeleNP Assessment Model
Since August 2020, we have implemented a model of TeleNP in our postgraduate program that can be used as a guide to the assessment of patients with congenital or acquired brain injury and in patients with neurodevelopmental, neurodegenerative, or other neuropsychological disorders (Fig. 2; S3 for Spanish). It begins when a specialist meets a patient or caregiver who requires neuropsychological evaluation or when a patient seeks such assessment on their own. The first contact is by telephone to determine whether the patient is a possible candidate for TeleNP. If so, they are sent an informed consent form and a link for the online platform for the first session, in which clinical data are taken and the technological resources that will be necessary for assessment are identified. Subsequent sessions consist of the administration of instruments with specific adaptations for their use in TeleNP. Finally, the report is prepared, and there is a session to provide the patient or caregiver with results, feedback, and possible recommendations for intervention or treatment. The referring physician is also be contacted with the results, where necessary, to discuss the treatment steps to be taken.
Referral by a Medical Specialist, Caregiver, or School
The referral for neuropsychological evaluation is generally from a medical specialist (neurosurgeon, neurologist, geneticist, or psychiatrist), parents, caregiver, or school. They are asked to refer patients with the appropriate characteristics for remote evaluation, excluding mainly those in which the severity of a neurological or psychiatric condition would preclude such evaluation. They are generally also asked to refer patients who are at least 6 years old, the age at which studies have reported that remote evaluation is appropriate (Harder et al., 2020; Ransom et al., 2020). Younger children can be evaluated when the child has the cognitive and behavioral characteristics (language comprehension, attention span of at least 30 minutes, without motor and sensory problems, and moderately understandable expressive language) that allow it. Another option for young children is to use developmental scales for which data can be obtained from a parent or caregiver. For the final decision, it is necessary to consider the individual characteristics of each case.
Telephone contact and checklist for TeleNP
Similar to other TeleNP models (Arias et al., 2020; Kitaigorodsky et al., 2021; Loman et al., 2021; Peterson, Ludwig, & Jashar, 2021; Ransom et al., 2020; Thibodaux et al., 2021), our first contact with a family member or in some cases the patient is by telephone. The evaluator asks a series of brief questions to determine whether the patient has appropriate support, as well as a computer, reliable internet, a printer, and the software needed to carry out the assessment (Table 2). Some of the checklist questions have previously been used in other institutions carrying out TeleNP in Latino populations (Arias et al., 2020). The family member is also asked about the patient’s health and whether they have a medical diagnosis, to learn about their condition and acquire an overview of its clinical manifestations before the evaluation. However, it is not necessary for the patient to already have or know their medical diagnosis; indeed, in some cases, the neuropsychological assessment may be helpful in determining it.
Table 2.
1. Does the patient have a computer or tablet with a camera? |
• If not, can they obtain one? |
2. Do they have an internet connection? |
3. Is their internet connection reliable? |
• Do they have the videoconferencing platform installed? |
• Do they have the resources to install it? |
4. Do they have a printer? |
• If not, can they print material for the evaluation sent electronically? |
6. Do they have email or an instant messaging application? |
If the family member indicates that they have everything necessary for remote evaluation, an informed consent form is sent by email or text message that will need to be signed by the family member and/or the patient and witnesses, and then scanned or photographed and returned to the evaluator. They are also sent a link to a secure platform on which the evaluation will be carried out, along with the date and time of each of the sessions. The first session is an interview to take a clinical history, and the necessary instruments are administered in subsequent sessions.
If the patient lacks the necessary equipment or the support needed to operate it, they can be asked about the possibility of borrowing a computer or using one in the home of someone they know; if not, they are put on the waiting list for a face-to-face assessment. However, in Mexico and other Latin American countries, it is possible that some patients and their families live far away or lack economic resources, such that traveling to a place where in-person assessments can be carried out is nearly impossible. In these cases, an experienced neuropsychologist must decide if it is possible to use instruments such as detailed semi-structured interviews, scales, and inventories over the telephone to at least formulate a diagnostic hypothesis, in order to have a follow-up psychoeducation session or provide the best recommendation possible.
Informed Consent
Informed consent for TeleNP must conform to APA guidelines (Harder et al., 2020; Postal et al., 2021; Wells et al., 2021): it must inform the patient about the protection of confidentiality, the procedure, objective, benefits, and risks of TeleNP evaluation, as well as the vulnerability of confidentiality in the use of a web platform. It is also suggested that a statement be included that the evaluator may ask to make video recordings of some tasks to reliably observe the patient’s performance, with the assurance that the recordings will later be destroyed (Thibodaux et al., 2021). In Mexico, similar requirements are specified in the Psychologist’s Ethical Code of the Mexican Psychological Society (Mexicana and de Psicología, 2010, SMP), particularly Articles 23, 35, 61, 62, and 118–124 (SMP, 2010). According to the recommendations of the working group for the practice of TeleNP in Latin America, consent must be obtained in accordance with the laws of each country; thus, it is essential for professionals to know the applicable laws in their countries and localities (Crivelli et al., 2021).
Conditions of the TeleNP service should be noted, and a checklist should be provided where the family member indicates that they have access to the essential tools for the evaluation, as well as a statement of approval for use of the data with protection for the patient’s confidentiality. Finally, the informed consent document should indicate that modifications will be made to the instruments for the purposes of TeleNP and that valuable qualitative information may therefore be lost (Bilder et al., 2020; Crivelli et al., 2021; Kitaigorodsky et al., 2021; Postal et al., 2021; SMP, 2010). An example in English and Spanish of an informed consent document is provided in Supplementary Material 2.
Interview and Clinical History and Screening for TeleNP Evaluation
The first decision to make regarding the use of TeleNP is the choice of an application or platform for the evaluation sessions. Although no specifications are given by the SMP or the Latin American working group for teleneuropsychology regarding specific platforms, the APA, the IOPC, and others (see Arias et al., 2020; Nguyen et al., 2021; Pérez et al., 2021; Scott, Marton, & Madore, 2022) suggest using an application that allows for adequate management and protection of data. In the United States, the recommendation is to use applications that meet the criteria of the Health Insurance Portability and Accountability Act (HIPAA; Arias et al., 2020; Bilder et al., 2020; Inter Organizational Practice Committee, 2020; Scott et al., 2022), which include Skype for Business®, Updox®, VSee®, Zoom®, Doxy.me®, Google G Suite Hangouts®, Cisco Webex Meetings®, Amazon Chime®, GoToMeeting®, Spruce Health Care Messenger®, and others (U.S. Department of Health & Human Services, 2021). Mexico has no law regulating these applications, but according to the IOPC, the professional version of Zoom meets HIPAA standards (Bilder et al., 2020), and 50% of the articles reviewed recommend its use for TeleNP. If Zoom is not available, other applications with data protection can be used. Pérez et al. (2021) provide a review of some other platforms with respect to their use in TeleNP.
Once a platform has been chosen, the patient, parent, or caregiver, depending on the case, should be sent the link that allows them to connect to the session at the established date and time. It is recommended that this be sent one to three days in advance. It is suggested that they be called on the telephone a day in advance to confirm the appointment, to check that the person in charge of setting up the equipment has the application installed on the device, and to answer any questions about its use. It is important to note that a low educational level, common in many Latin American populations, may correlate with low technological literacy, which could interfere with the use of technological resources (Crivelli et al., 2021). Clinicians should ensure that patients and their caregivers have sufficient command of technological resources during the first session; otherwise, the presence of a facilitator may be needed during key moments of the assessment. In this first session, as in the model of Thibodaux et al. (2021), and similar to the procedure in a face-to-face assessment, the interview should review the clinical history of the patient, along with a second checklist (Table 3) related to the physical environment in which the evaluation will be carried out and to confirm that the patient is indeed a candidate for remote assessment (Crivelli et al., 2021; Ransom et al., 2020). An inappropriate physical environment includes a television or radio near the computer, its location in a living room where other people might be present, or inadequate lighting. In developing countries, including Mexico and others in Latin America, many households unfortunately have their computers in the living room because they lack a specific work area; one recommendation is that they turn off the radio and television during the assessment and use headphones. Ideally, no other person except the patient and, if necessary, the person providing assistance should be present in the same space during sessions. If this is not possible, the patient should be put on the waiting list for a face-to-face assessment. When carrying out the assessment of a legally independent adult, if the assistance of a third person is necessary, the SMP ethical guidelines on confidentiality require the patient’s consent (SMP, 2010).
Table 3.
1. Does the patient have the ability to manage and use the device and platform? |
• If not, is there someone who can assist them during the evaluation sessions? |
2. Is the patient’s physical environment appropriate for remote evaluation? |
3. Was the informed consent signed and returned to the evaluator? |
4. Is the parent or guardian, and where possible the patient, clear about what will be done with TeleNP? |
Assessment and Adaptations with TeleNP
This section describes the procedure before, during, and after evaluation sessions, as well as adaptations that must be made to certain tasks.
Before the evaluation (preparation of equipment, physical space, and printed materials)
It is advisable to review the following checklist before beginning a TeleNP session.
Verify that the patient is using a tablet, laptop, or desktop computer. Although TeleNP can be carried out with any type of device (smartphone, tablet, or computer), it is preferable that the screen be at least seven inches (17 cm) in width (Alegret et al., 2021; Cullum et al., 2014; Galusha-Glasscock, Horton, Weiner, & Cullum, 2016; Gnassounou et al., 2021; Harder et al., 2020; Kitaigorodsky et al., 2021; Nussbaum et al., 2021; Palmese et al., 2022; Peterson et al., 2021); the devices of choice are a desktop, laptop, or large tablet computer (Pritchard, Sweeney, Salorio, & Jacobson, 2020). Some authors suggest that evaluation of children older than 3 is more effective with a desktop or laptop, but a large percentage of families in Mexico do not have these. More than 50%, however, have a smartphone or tablet to connect to the internet (Instituto Nacional de Estadística y Geografía, 2021), so these could be used. Although the use of smartphones is not recommended, since it could affect the size and presentation of standardized visual material with the shared screen option (Loman et al., 2021), some authors nevertheless report that they can be used (Alegret et al., 2021; Pulsifer et al., 2021). Conditions in Mexico and Latin America often do not meet the ideals established in other parts of the world, so it is necessary to analyze the advantages and disadvantages of existing conditions case by case.
In Mexico and Latin America, there are often problems with image quality because of low bandwidth (Crivelli et al., 2021). Some authors (Franco-Martin, Bernardo-Ramos, & Soto-Pérez, 2012) recommend scheduling sessions at times with greater bandwidth to improve the quality of the video call. When internet service is of low quality, assessment should be limited to screening tests (e.g., MoCA Test, Nasreddine et al., 2005; DP-3, Alpern, 2018; CUMANIN-2, Portellano, Mateos, Martínez Arias, & Sánchez-Sánchez, 2021; CUMANES, Portellano, Mateos, & Martínez Arias, 2022; Battelle screening, Newborg, Stock, & Wnek, 2011), tests without timed responses (e.g., memory and verbal tests), or inventories administered by telephone (Crivelli et al., 2021); e.g., ABAS-II (Harrison & Oakland, 2021), SENA (Fernández-Pinto, Santamaría, Sánchez-Sánchez, Carrasco, & del Barrio, 2015), BRIEF-2 (Gioia, Isquith, Guy, & Kenworthy, 2017), and BRIEF-A (Roth, Isquith & Gioia, 2005).
Verify the functionality of the camera and microphone of both the patient and the evaluator (Tailby et al., 2020). It is important to ensure that they can see and hear one another and can use the share screen option, since it will be necessary for the online evaluation (Koterba et al., 2020; Postal et al., 2021).
The patient must be in an isolated space with a minimum of noise, with sufficient light, and seated at a table. It is important to reduce to a minimum factors that might influence the patient’s performance, including the presence of external stressors, distractors, or help (Koterba et al., 2020; Parlar et al., 2020; Ransom et al., 2020; Thibodaux et al., 2021). If possible, it is recommended that the patient be in a room with the door closed and with a “Do Not Disturb” sign (Hewitt, Rodgin, Loring, Pritchard, & Jacobson, 2020) and that the person assisting them be outside of their visual field.
The patient should have the printed material that will be used in the session (Thibodaux et al., 2021). The evaluator should send it 24–48 hours before the session. According to Arias et al. (2020) and Tailby et al. (2020), printed materials should be in a closed folder or envelope on the table where the patient is seated, and the patient should not see them before the evaluation.
It is important to check that the patient has the necessary writing utensils (two blunt-tipped pencils and a sharpener) for the graphomotor tests. Other tools will generally not be needed for the assessment. We suggest not using tests that require specific materials (e.g., Corsi cubes), as it is complicated to send them to the patient’s home.
The camera should focus on the patient, making it possible to see their hands, shoulders, and head (Mahon, Webb, Snell, & Theadom, 2021; Postal et al., 2021).
Although the patient may be adept in the use of the computer and videoconferencing software, a third person may need to be present to make recordings of the performance of tasks, to corroborate certain information provided by the patient, and to help in case of technical difficulty (Parsons et al., 2021). Except in such cases, the evaluation should be performed with the patient in private (Loman et al., 2021).
If the presence of another person is necessary, the evaluator should have that person’s telephone number in case further communication is necessary (Loman et al., 2021; Tailby et al., 2020). It is important to emphasize that the person assisting the patient should not help them with the evaluation tasks.
Occasionally, some instruments may be discarded or invalidated due to environmental factors (e.g., external distractors, poor skills in using the platform, major cognitive alterations). It is recommended that evaluators have alternative tests for the assessment.
During the Assessment
During the sessions, the evaluator should do the following:
Introduce providers and trainees who are on the call, allowing time to create sufficient rapport with the patient so that the evaluation can be carried out well. Questions about personal information, hobbies, favorite games, or television programs can help create rapport.
Offer recommendations and explain the basic functions of the platform.
Confirm that the patient has the folder or envelope with the printed material and that they have not yet looked at it (Thibodaux et al., 2021).
Be sure that the patient has the computer or device fully charged, with the charger available, or connected to an outlet (Nussbaum et al., 2021; Thibodaux et al., 2021).
If the patient is accompanied, explain that the only role of the parent or caregiver is to assist the patient if there is a technical or technological problem. Emphasize that they should not help the patient perform the tasks or answer the questions on the tests.
Explain that there should be no interruptions, except between subtests.
For reasons of confidentiality and copyright, patients should not take photographs, videos, or screen captures of the material projected (Loman et al., 2021; Postal et al., 2021); the evaluator will make recordings when they are needed.
Explain that the material sent for printing is exclusively for use in the evaluation and that it must not be reproduced for sale or any other reason (Hewitt et al., 2020; Pritchard et al., 2020). After it is printed, the email in which it was sent must be deleted.
Verify before beginning the session that the printed material is of sufficient quality to be used (legible, appropriate margins, correct page orientation).
Emphasize, to reduce any stress over the use of the technology, that the evaluator is willing to answer any questions about its use.
Record all the patient’s responses on paper, according to proper protocol, to be scored and stored later, following normal procedures for the protection of data (Loman et al., 2021).
View the patient with the largest image possible (i.e., configure the video with the patient in speaker mode) to better observe their performance and behavior (Hewitt et al., 2020). For certain tasks, the patient can be asked to adjust the camera angle or even record the task from another device to evaluate performance from different angles (Pritchard et al., 2020).
Share digital stimuli in full-screen mode to avoid distracting the patient with other content on the screen (Loman et al., 2021; Thibodaux et al., 2021).
For the tests using pencil and paper, the patient can be asked to record the execution of tasks or join the videoconference from another device focused on their performance. Mahon et al. (2021) suggest that this second camera shows a side view.
Be sensitive to emotional and attentional changes in the patient that can affect their performance. Some authors suggest adjusting the cognitive load of the evaluation to the clinical condition of the patient, with priority given to sustaining attention (Sozzi et al., 2020; Thibodaux et al., 2021). If significant fluctuations are seen, it may be necessary to take breaks during the session or end it and continue evaluation in a later session.
At the End of the Session
The following steps are recommended at the end of a TeleNP session:
The patient or their parent or caregiver should send scans or photos of their work, preferably immediately after the session, so that they do not forget or lose material.
Patients should delete photos and destroy work on paper as soon as they have been sent to the evaluator.
Printed material not used should be saved in a closed folder or envelope, since it will be used in the next session.
The evaluator should indicate to the patient, parent, or guardian which materials will be necessary for the next session.
The evaluator should end the videoconference and create a link for the next session.
It is recommended that the assessment sessions last 60–90 minutes (Fox-Fuller et al., 2021; Galusha-Glasscock et al., 2016; Kitaigorodsky et al., 2021; Parks, Davis, Spresser, Stroescu, & Ecklund-Johnson, 2021).
Adaptations for TeleNP
In addition to the recommendations described, it is important to adjust the evaluation to ensure greater reliability and fewer unknowns that might influence the patient’s performance. As suggested by Thibodaux et al. (2021), where possible, there should be at least one expert evaluator in neuropsychology and one in training. The former should be the principal evaluator, and the latter can provide support. The principal evaluator should conduct the session and carry out all the steps we have described. The second evaluator can take over if there are technical difficulties, but if one is not available, a single expert evaluator should manage the evaluation, take care of any technical difficulties that may arise, and follow the suggestions we have made.
We recommend certain strategies for specific tasks:
For graphomotor tasks, the accompanying parent or caregiver can be asked to connect with an additional device or record the execution of the task with another camera, in addition to the camera focused directly on the patient, to assess their performance.
In some working memory tasks, such as digit span or mental calculation, the patient can be asked to raise their hands to avoid having them make notes of the stimuli or use another device to improve their performance (Hewitt et al., 2020).
We also suggest keeping in mind the following points during the administration of the tests:
Instruction in the use of the remote control. The evaluation of some processes, such as following instructions or naming tasks (e.g., visual-verbal or visual-graphic naming), requires that the patient controls the shared content from their computer to indicate visual stimuli and make notes about them. The Zoom platform allows the viewer to control the cursor and make notes remotely, but both the evaluator, as host of the video meeting, and the patient must activate this feature. It is recommended that a practice activity be carried out so that the patient can familiarize themselves with the use of this feature and minimize the probability of technical difficulties that could interfere with the rhythm of the session and their performance on the tasks. The evaluator should ascertain before the session begins that the patient understands its use and provide any needed guidance.
Breaks during the evaluation. It is recommended to have a 5–10 minutes break that can be used halfway through the administration of the instruments to give the patient a cognitive rest (Parsons et al., 2021). This can include physical activity, such as stretching the arms and neck (rotating or forward and back motion), walking around the room, or other activities such as looking at images, music videos, or scenes from movies or cartoons that the patient would enjoy.
Use of behavioral strategies. The evaluator can use various strategies to encourage the patient’s cooperation throughout the session. Reinforcement techniques or a point system can easily be applied during TeleNP: after finishing a subtest, the patient can be given a token or shown a sticker, and after completing a number of tasks, they can be allowed a fun activity (Amador-Salinas, Rivera, & Ojeda, 2020). If a patient shows serious behavioral difficulties that affect their completion of the evaluation, it may require the participation of a caregiver who can sit next to them, help keep their attention on the task, and encourage them to do it (Hewitt et al., 2020). This should be done only in exceptional cases after having tried other strategies.
Scanning of tests. It will be necessary to scan some tests to be used on a shared screen, taking care not to make copies that violate copyright. It is recommended that the test material be checked beforehand to ensure that it can be adapted for presentation on a screen (Koterba et al., 2020). The size and orientation of the printed page should be respected; transitions or animations can be used to hide portions of tasks that require a specific exposure time and to avoid distractions by showing just the portion used. It is important to check that the material is not cut off, blurry, badly oriented, or significantly different in size (Hewitt et al., 2020). Patients should also be told during the informed consent process that the material must not be reproduced.
Administration of instruments to informants. It is possible to solicit a behavioral or emotional evaluation of the patient from one or more informants (primary caregivers, family members, and/or teachers) to identify their behavior in daily life and ascertain whether they show clinical signs in contexts outside the evaluation, such as at home or school. Such data can be of great use in complementing a diagnosis (Hewitt et al., 2020; Sozzi et al., 2020). The forms for the scales should be scanned and sent electronically to the informant, who can return them to the evaluator either in printed or scanned form, following the measures taken to insure exclusively clinical use (Hewitt et al., 2020). These can also be sent using the digital platforms of the instruments themselves.
Preparation of the TeleNP Evaluation Report
The TeleNP evaluation report, such as the report of a face-to-face assessment, should include patient identifying information, the reason for consultation, important data obtained from the clinical history, justification for the tests administered, results, neuropsychological findings, diagnostic conclusion, and recommendations. One important difference from the face-to-face report is that the section on the tests applied should explain the adaptations made for the TeleNP evaluation and report that consent for a TeleNP modality was obtained (Kitaigorodsky et al., 2021; Parlar et al., 2020). Another difference is that the evaluator should consider the uncontrolled factors associated with remote evaluation that could affect the reliability of the results, such as problems with the internet connection, interruptions, distractions in the patient’s location, inadequate workspace, or other persons nearby (Hewitt et al., 2020; Loman et al., 2021), difficulties that are often present in TeleNP (Parsons et al., 2021; Ransom et al., 2020). Finally, it should be noted that the results in the report should be taken with caution and that a face-to-face reevaluation should be carried out in 6–12 months (Hewitt et al., 2020; Koterba et al., 2020; Parlar et al., 2020; Ransom et al., 2020; Thibodaux et al., 2021).
Delivery of the TeleNP Report
As with face-to-face assessment, the results of a TeleNP evaluation should be provided to the patient and/or parent or caregiver with an explanation and the diagnostic conclusion, as well as brief psychoeducation about the diagnosis and recommendations for follow-up. It is recommended that the report be provided on a shared screen so that the parent or caregiver can visualize and follow the findings as they are explained (Bilder et al., 2020). The discussion can begin by reviewing the reason for consultation, explaining the structure of the report, the clinical history, the instruments used, and the results found with each test. It is important to explain the neuropsychological findings and provide a diagnosis or conclusions and recommendations. It is also important to suggest a face-to-face assessment as soon as conditions permit it (Thibodaux et al., 2021).
In Mexico, most neuropsychological assessments are at private clinics, and some patients decide to pursue private multidisciplinary follow-up. In these cases, the neuropsychologist should explain the findings and diagnosis to the patient, their parents, or caregivers so that they can explain their condition to other health or education professionals, and a copy of the report should be sent to medical or other pertinent health professionals.
Limitations of TeleNP
TeleNP as a means of providing neuropsychological services has many advantages but also limitations, as described below:
Difficulties in managing the behavior of young children or of patients with serious behavioral problems, especially those including impulsivity and attention problems (Pritchard et al., 2020).
Less control of measurement errors in the administration of tasks evaluating language, visuomotor abilities, and processing speed, which are also particularly susceptible to technical problems (Loman et al., 2021).
There are few standardized instruments for online administration, or rather, there is no standardization of the modifications made for TeleNP administration. There are few tests or instruments that have been validated for remote administration (Marra, Hamlet, Bauer, & Bowers, 2020), which makes it difficult to reliably evaluate all of the domains of cognitive function using TeleNP (Hewitt et al., 2020). In particular, to our knowledge, no such instruments have been validated in Mexico or Latin American countries. Clinical experience and environmental control have a major relevance for TeleNP assessment and diagnosis.
Uncontrolled environmental distractors influence the patient’s performance (Hewitt et al., 2020).
Remote interaction limits the precision of qualitative observations of the patient’s affective, cognitive, and behavioral characteristics, including their social abilities, which could have implications for diagnosis (Sozzi et al., 2020).
Although TeleNP appears to be an accessible option for many patients, a segment of the population with socioeconomic or other limitations may not have access to videoconferencing technology (Pritchard et al., 2020; Ransom et al., 2020). Internet usage varies widely across Latin American and Caribbean populations: only 12%–25% in Haiti and Nicaragua, 51%–54% in Paraguay and Panama, 62%–64% in Colombia and Mexico, and 70%–80% in Argentina, Chile, the Bahamas, and Barbados (Pierce et al., 2021).
Finally, the ethical codes in Latin American countries are a limitation, since they include no specific norms for the practice of TeleNP (Caicedo-Guale, Vera-García, & Sornoza-Palma, 2020).
In addition to these general limitations associated with TeleNP, it is important to consider that there are no Latin American guidelines or studies, either randomized or nonrandomized, that systematically assess this method of evaluation. However, the model proposed here should be considered an option based on theoretical foundations, one that has been carried out in practice in the Neuropsychology Residency Program at the FES-I, UNAM. The experience of the past 2 years suggests that it is feasible and useful, and its reliability and validity may be evaluated in future research.
Future Directions
With the COVID-19 pandemic of the past 2 years, research on TeleNP evaluation and its use has gained importance. The suspension of in-person activities has quickly led to the use of online assessment methods. Although different studies and guidelines have shown that this type of evaluation is reliable and that its results are similar to those obtained face-to-face, its clinical and basic application should be studied further. Singh and Germine (2021) use the term “hybrid neuropsychology” for a new model that makes use of a neuropsychologist’s in-person and online abilities. However, there is a need to create, standardize, and normalize online instruments to corroborate the reliability of TeleNP evaluation in diverse clinical samples and to carry out controlled, randomized trials to assess the effectiveness of intervention programs. Although face-to-face activities should not be eliminated, TeleNP can help to complement the activities of clinical neuropsychology. Technological limitations specific to developing countries still need to be addressed in Latin America for TeleNP and other telemedicine services to be available to populations in need.
Conclusion
The rapid transition to remote clinical practice motivated by the COVID-19 pandemic has created many challenges but has also required clinicians to overcome many of the limitations of TeleNP practice, allowing for unprecedented development in the field (Pritchard et al., 2020).
It is important to keep in mind that developing countries face limitations regarding living conditions and access to health and technological services, and thus encounter challenges beyond those experienced by high-income countries in the application of new clinical models and technologies. In general, Latin American infrastructure for mental health care is lacking in resources, particularly mental health care providers and training programs (Kohn et al., 2018). This model could thus be used to provide neuropsychology services to places where access is complicated and health personnel are scarce, but where internet access is available.
Although this model was applied in Mexico, there are similarities between Mexico and the rest of Latin America that make it possible to use it in other countries. The increase in mobile telephone ownership in the region with respect to the previous decade and the more modest increase in computer use (OECD & World Bank, 2020; Quiroga-Parra et al., 2017) facilitate its use, but the use of smartphones for TeleNP assessment in Latin America should not be ruled out.
Another challenge faced by practitioners in Mexico and other Latin American countries is the lack of digital platform norms for psychotherapy and neuropsychological procedures and the difficulty in finding a secure platform. Our model proposes the use of Zoom, an economically accessible platform that meets the norms of other countries for the care of personal data and secure access. We also found that patients and caregivers in other countries seem to have greater knowledge about the use of digital platforms; in our case, educational sessions had to be held on the use of the platform prior to the administration of tests. Most of the studies, we reviewed identified adequate neuropsychological testing and diagnostic reliability in clinic-to-clinic models. In our case, however, both the neuropsychologist and the patient are in their respective homes, so we sought strategies to control environmental factors that could affect the assessment. Most studies of reliability in assessing children addressed the normal population, but in recent years, there has been a need for online assessment of patients with neurodevelopmental and neurodegenerative disorders and acquired brain injury. The adequacy of instructions and screen-sharing features of digital applications was thus important. Although most of the literature confirms the validity and reliability of TeleNP, we found it necessary to implement strategies, adaptations, and alternatives for neuropsychological assessment in our clinical practice. Among the studies we reviewed, only that of Fox-Fuller et al. (2021) investigated a home-to-home model; it found adequate test–retest reliability. Some of their suggested strategies, such as testing time and care with environmental factors, could be applied in our model; we have also added others, such as the type of electronic device and care with environmental factors in the homes of both the patient and the neuropsychologist, that are specific to Latin America.
Latin American countries have ethical codes adapted to their local contexts. All of these include confidentiality guidelines (Caicedo-Guale et al., 2020), but in the case of TeleNP, some confidentiality requirements must be modified, which may increase vulnerability. In the development and practice of this model for the TeleNP assessment, we identified the importance of creating new norms for online clinical practice and confidentiality.
There is a lack of studies and guidelines in the literature addressing the circumstances of Mexico and other Latin American countries, and some of the recommendations in the literature are not applicable to this region. However, circumstances such as the COVID-19 pandemic highlight the need for TeleNP in order to care remotely for pediatric and adult populations with different neuropsychological pathologies. The Residency in Clinical Neuropsychology of the Master’s and Doctoral Programs in Psychology of the Faculty of Psychology, FES-I, UNAM has implemented the model described here over the past year, based on the recommendations of international institutions, a literature review, and the clinical experience of program faculty. Although the validity of the model has not yet been confirmed, the suggestions and procedure described here can be used as a provisional guide for clinical practice and as a basis for future research. However, it is the clinician’s responsibility to decide when it is appropriate to use this method, prioritizing the clinical needs and benefits of each patient (Ransom et al., 2020). If there are only marginal benefits, or risks associated with its use, the suggestion is to use face-to-face assessment.
According to some authors (Pérez et al., 2021; Sherwood & MacDonald, 2020; Wadsworth et al., 2018), patients with mild traumatic brain injury, symptoms of cognitive deterioration, emotional or behavioral changes, and those recovering from neurological illness or at risk for cognitive effects of medication are some of those for whom TeleNP has been demonstrated to be useful for evaluation, diagnosis, and follow-up.
Although the long-term reliability of TeleNP has yet to be confirmed, studies to date (Hewitt et al., 2020; Loman et al., 2021; Parsons et al., 2021; Ransom et al., 2020) suggest that it is a viable and accessible method for providing neuropsychological services in the context of social distancing and that it also has the potential to broaden access to these services. Research on its strengths and limitations is thus a growing area of interest. The clinic-to-clinic and clinic-to-home models have received the greatest attention, but the study by Fox-Fuller et al. (2021) has shown that the home-to-home model of TeleNP evaluation also has appropriate reliability. Our proposal can also be used after the pandemic with clinic-to-home and home-to-home models to reach populations that are far from medical centers. Our suggestions are not a substitute for face-to-face clinical practice, and they should not be taken as requirements for evaluation. However, they provide a model for clinical activity whose validity and reliability can be tested in randomized clinical studies.
Conflict of Interest
The authors report that there are no competing interests on declare.
Supplementary Material
Contributor Information
Carolina Reyes-Méndez, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
Belén Prieto-Corona, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
Ana Natalia Seubert-Ravelo, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
Julieta Moreno-Villagómez, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
José-Ángel Cabañas-Tinajero, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
Ma Guillermina Yáñez-Téllez, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
Rocío Alejandra Quezada-Torres, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
Marybeth Téllez-Rodríguez, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
Bárbara Barrera-Rodríguez, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
Martha Paola Soto-Jiménez, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
Fátima Aideé González-Gutiérrez, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
Elena Castillo-Tejeda, Residencia de Neuropsicología Clínica, División de Investigación y Posgrado, Facultad de Estudios Superiores Iztacala, UNAM, Tlalnepantla de Baz, México.
References
- Alegret, M., Espinosa, A., Ortega, G., Pérez-Cordón, A., Sanabria, Á., Hernández, I., et al. (2021). From face-to-face to home-to-home: Validity of a Teleneuropsychological battery. Journal of Alzheimer’s Disease, 81 (4), 1541–1553. PubMed. 10.3233/JAD-201389. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Alpern, G. D. (2018). DP-3. Perfil de Desarrollo-3 (F. Sánchez-Sánchez, adaptación española). Madrid: TEA Ediciones. [Google Scholar]
- Amador-Salinas, J. G., Rivera, V. G., & Ojeda, P. L. (2020). Incremento en la entrega de tareas escolares por medio de la economía de fichas grupal. Revista Digital Internacional de Psicología y Ciencia Social, 6(2), 372–387. 10.22402/j.rdipycs.unam.6.2.2020.282.372-387. [DOI] [Google Scholar]
- American Psychological Association (2013). Guidelines for the practice of telepsychology. American Psychologist, 68(9), 791–800. 10.1037/a0035001. [DOI] [PubMed] [Google Scholar]
- Arias, F., Safi, D. E., Miranda, M., Carrión, C. I., Diaz Santos, A. L., & Armendariz, V. (2020). Teleneuropsychology for monolingual and bilingual Spanish-speaking adults in the time of COVID-19: Rationale, professional considerations, and resources. Archives of Clinical Neuropsychology, 35(8), 1249–1265. 10.1093/arclin/acaa100. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bilder, R. M., Postal, K. S., Barisa, M., Aase, D. M., Cullum, C. M., Gillaspy, S. R.et al. (2020). InterOrganizational practice committee recommendations/guidance for teleneuropsychology (TeleNP) in response to the COVID-19 pandemic. The Clinical Neuropsychologist, 34(7–8), 1314–1334. 10.1080/13854046.2020.1767214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Brearly, T. W., Shura, R. D., Martindale, S. L., Lazowski, R. A., Luxton, D. D., Shenal, B. V.et al. (2017). Neuropsychological test administration by videoconference: a systematic review and meta-analysis. Neuropsychology Review, 27(2), 174–186. 10.1007/s11065-017-9349-1. [DOI] [PubMed] [Google Scholar]
- Caicedo-Guale, L. C., Vera-García, M., & Sornoza-Palma, W. K. (2020). CÓDIGO DE ÉTICA DE LOS PAÍSES LATINOAMERICANOS: Artículo de investigación. Revista Científica Multidisciplinaria Arbitrada Yachasun, 4(7 Ed. esp.), 102–110. 10.46296/yc.v4i7edesp.0063. [DOI] [Google Scholar]
- Chapman, J. E., Gardner, B., Ponsford, J., Cadilhac, D. A., & Stolwyk, R. J. (2021). Comparing performance across in-person and videoconference-based administrations of common neuropsychological measures in community-based survivors of stroke. Journal of the International Neuropsychological Society, 27(7), 697–710. Cambridge Core 10.1017/S1355617720001174. [DOI] [PubMed] [Google Scholar]
- Crivelli, L., Quiroz, Y. T., Calandri, I. L., Martin, M. E., Velilla, L. M., Cusicanqui, M. I.et al. (2021). Working group recommendations for the practice of Teleneuropsychology in Latin America. Archives of Clinical Neuropsychology, 37(3), 553–567. acab080 PubMed 10.1093/arclin/acab080. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cullum, C. M., Hammers, D. B., Harder, L., & Stolwyk, R. (2020). Teleneuropsychology (TeleNP) in response to COVID-19. International Neuropsychological Society. https://www.the-ins.org/webinars/.
- Cullum, C. M., Hynan, L. S., Grosch, M., Parikh, M., & Weiner, M. F. (2014). Teleneuropsychology: Evidence for video teleconference-based neuropsychological assessment. Journal of the International Neuropsychological Society, 20(10), 1028–1033.Cambridge Core 10.1017/S1355617714000873. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fernández-Pinto, I., Santamaría, P., Sánchez-Sánchez, F., Carrasco, M. A., & delBarrio, V. (2015). SENA. Sistema de evaluación de niños y adolescentes. Madrid: TEA Ediciones. [Google Scholar]
- Fox-Fuller, J. T., Ngo, J., Pluim, C. F., Kaplan, R. I., Kim, D.-H., Anzai, J. A. U.et al. (2021). Initial investigation of test-retest reliability of home-to-home teleneuropsychological assessment in healthy, English-speaking adults. The Clinical Neuropsychologist, 1–15. 10.1080/13854046.2021.1954244. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Franco-Martin, M. A., Bernardo-Ramos, M., & Soto-Pérez, F. (2012). Ciber-Neuropsicología: Aplicación de nuevas tecnologías en la evaluación neuropsicológica. Actas Españolas de Psiquiatría, 40(6), 308–314. [PubMed] [Google Scholar]
- Galusha-Glasscock, J. M., Horton, D. K., Weiner, M. F., & Cullum, C. M. (2016). Video teleconference Administration of the Repeatable Battery for the assessment of neuropsychological status. Archives of Clinical Neuropsychology, 31(1), 8–11. PubMed. 10.1093/arclin/acv058. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gioia, G. A., Isquith, P. K., Guy, S. C., & Kenworthy, L. (2017). In Maldonado, M. J., Fournier, M. C., Martínez-Arias, R., González-Marqués, J., Espejo-Saavedra, J. M., Santamaría, P.et al. (Eds.), BRIEF-2. Evaluación conductual de la función ejecutiva. Madrid: TEA Ediciones. [Google Scholar]
- Gnassounou, R., Defontaines, B., Denolle, S., Brun, S., Germain, R., Schwartz, D.et al. (2021). Comparison of neuropsychological assessment by videoconference and face to face. Journal of the International Neuropsychological Society, 28(5), 1–11. Cambridge Core 10.1017/S1355617721000679. [DOI] [PubMed] [Google Scholar]
- Grosch, M. C., Gottlieb, M. C., & Cullum, C. M. (2011). Initial practice recommendations for Teleneuropsychology. The Clinical Neuropsychologist, 25(7), 1119–1133. 10.1080/13854046.2011.609840. [DOI] [PubMed] [Google Scholar]
- Harder, L., Hernandez, A., Hague, C., Neumann, J., McCreary, M., Cullum, C. M.et al. (2020). Home-based pediatric teleneuropsychology: a validation study. Archives of Clinical Neuropsychology, 35(8), 1266–1275. 10.1093/arclin/acaa070. [DOI] [PubMed] [Google Scholar]
- Harrison, P. L., & Oakland, T. (2021). In Centeno, D. M., & Fernández-Pinto, I. (Eds.), adaptación españolaABAS-II. Sistema Para la evaluación de la conducta adaptativa. Madrid: TEA Ediciones. [Google Scholar]
- Hewitt, K. C., Rodgin, S., Loring, D. W., Pritchard, A. E., & Jacobson, L. A. (2020). Transitioning to telehealth neuropsychology service: considerations across adult and pediatric care settings. The Clinical Neuropsychologist, 34(7–8), 1335–1351. 10.1080/13854046.2020.1811891. [DOI] [PubMed] [Google Scholar]
- Inter Organizational Practice Commitee . (2020). Evidence based neuropsychological care during the COVID-19 pandemic. https://iopc.online/teleneuropsychology.
- Kirkwood, K. T., Peck, D. F., & Bennie, L. (2000). The consistency of neuropsychological assessments performed via telecommunication and face to face. Journal of Telemedicine and Telecare, 6(3), 147–151. 10.1258/1357633001935239. [DOI] [PubMed] [Google Scholar]
- Kitaigorodsky, M., Loewenstein, D., Curiel Cid, R., Crocco, E., Gorman, K., & González-Jiménez, C. (2021). A Teleneuropsychology protocol for the cognitive assessment of older adults during COVID-19. Frontiers in Psychology, 12, 651136–651136. 10.3389/fpsyg.2021.651136. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kohn, R., Ali, A. A., Puac-Polanco, V., Figueroa, C., López-Soto, V., Morgan, K.et al. (2018). Mental health in the Americas: An overview of the treatment gap. Revista panamericana de salud publica, 42(e165), 1–10. 10.26633/RPSP.2018.165. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Koterba, C. H., Baum, K. T., Hamner, T., Busch, T. A., Davis, K. C., Tlustos-Carter, S.et al. (2020). COVID-19 issues related to pediatric neuropsychology and inpatient rehabilitation – Challenges to usual care and solutions during the pandemic. The Clinical Neuropsychologist, 34(7–8), 1380–1394. 10.1080/13854046.2020.1811892. [DOI] [PubMed] [Google Scholar]
- Loman, M., Vogt, E., Miller, L., Landsman, R., Duong, P., Kasten, J.et al. (2021). “How to” operate a pediatric neuropsychology practice during the COVID-19 pandemic: Real tips from one practice’s experience. Child Neuropsychology, 27(2), 251–279. 10.1080/09297049.2020.1830962. [DOI] [PubMed] [Google Scholar]
- Mahon, S., Webb, J., Snell, D., & Theadom, A. (2021). Feasibility of administering the WAIS-IV using a home-based telehealth videoconferencing model. The Clinical Neuropsychologist, 36(3), 1–13. 10.1080/13854046.2021.1985172. [DOI] [PubMed] [Google Scholar]
- Marra, D. E., Hamlet, K. M., Bauer, R. M., & Bowers, D. (2020). Validity of teleneuropsychology for older adults in response to COVID-19: A systematic and critical review. The Clinical Neuropsychologist, 34(7–8), 1411–1452. 10.1080/13854046.2020.1769192. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Banco Mundial (2021). Población Urbana en Latinoamérica y el Caribe. https://datos.bancomundial.org/indicador/SP.URB.TOTL.IN.ZS?locations=ZJ-MX. [Google Scholar]
- Nasreddine, Z. S., Phillips, N. A., Bédirian, V., Charbonneau, S., Whitehead, V., Collin, I.et al. (2005). The Montreal cognitive assessment, MoCA: a brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society, 53, 695–699 10.1111/j.1532-5415.2005.53221.x. [DOI] [PubMed] [Google Scholar]
- Newborg, J., Stock, J. R., & Wnek, L. (2011). In de laCruz López, M. V., & Criado, M. G. (Eds.), adaptación españolaBattelle. Inventario de desarrollo. Madird: TEA Ediciones. [Google Scholar]
- Nguyen, C. M., Tan, A., Nguyen, A., Lee, G. J., Qi, W., Thaler, N. S.et al. (2021). Cross-cultural considerations for teleneuropsychology with Asian patients. The Clinical Neuropsychologist, 1–15. 10.1080/13854046.2021.1948104. [DOI] [PubMed] [Google Scholar]
- Instituto Nacional de Estadística y Geografía, INEGI (2021). Encuesta Nacional sobre Disponibilidad y Uso de Tecnologías de la Información en los Hogares. México. https://www.inegi.org.mx/programas/dutih/2021/.
- Nussbaum, N. L., Young, S. R., DeLeon, R. C., Engelmann, M. L., & Schraegle, W. A. (2021). The future is now: pediatric neuropsychological presurgical epilepsy evaluation in the age of COVID-19. Epileptic Disorders: International Epilepsy Journal with Videotape, 23 (2), 274–280. PubMed. 10.1684/epd.2021.1274 [DOI] [PMC free article] [PubMed] [Google Scholar]
- OECD & World Bank . (2020). Panorama de la Salud: Latinoamérica y el Caribe 2020. OECD Publishing, Paris. 10.1787/740f9640-es [DOI] [Google Scholar]
- Oxford Centre for Evidence-Based Medicine (2011). Oxford Centre for Evidence-Based Medicine 2011 levels of evidence. The Centre for Evidence-Based Medicine. http://www.cebm.net/index.aspx?o=5653.
- Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D.et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLoS Medicine, 18(3), e1003583. 10.1371/journal.pmed.1003583. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Palmese, C. A., Wyman-Chick, K. A., Racine, C., Pollak, L. E., Lin, G., Farace, E.et al. (2022). Assessment of deep brain stimulation candidacy during the COVID-19 pandemic: lessons learned and future directions for neuropsychologists. The Clinical Neuropsychologist, 36(1), 72–84. 10.1080/13854046.2021.1929496. [DOI] [PubMed] [Google Scholar]
- Parks, A. C., Davis, J., Spresser, C. D., Stroescu, I., & Ecklund-Johnson, E. (2021). Validity of in-home teleneuropsychological testing in the wake of COVID-19. Archives of Clinical Neuropsychology, 36 (6), acab002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Parlar, M. E., Spilka, M. J., Wong Gonzalez, D., Ballantyne, E. C., Dool, C., Gojmerac, C.et al. (2020). “You can’t touch this”: Delivery of inpatient neuropsychological assessment in the era of COVID-19 and beyond. The Clinical Neuropsychologist, 34(7–8), 1395–1410. 10.1080/13854046.2020.1810324. [DOI] [PubMed] [Google Scholar]
- Parsons, M. W., Gardner, M. M., Sherman, J. C., Pasquariello, K., Grieco, J. A., Kay, C. D.et al. (2021). Feasibility and acceptance of direct-to-home tele-neuropsychology services during the COVID-19 pandemic. Journal of the International Neuropsychological Society, 28(2), 1–6. Cambridge Core 10.1017/S1355617721000436. [DOI] [PubMed] [Google Scholar]
- Pérez, P. K., Ramos Usuga, D., & Arango Lasprilla, J. C. (2021). Teleneuropsicología en países de habla hispana: Una mirada crítica al uso de Tecnologías de Información y Comunicación en la evaluación neuropsicológica. Revista Iberoamericana de Neuropsicología, 4(1), 1–27. [Google Scholar]
- Peterson, R. K., Ludwig, N. N., & Jashar, D. T. (2021). A case series illustrating the implementation of a novel tele-neuropsychology service model during COVID-19 for children with complex medical and neurodevelopmental conditions: a companion to Pritchard et al., 2020. The Clinical Neuropsychologist, 35(1), 99–114. 10.1080/13854046.2020.1799075. [DOI] [PubMed] [Google Scholar]
- Pierce, W., Schroeder, D., & Suchecki, R. (2021). Telehealth in Latin America: Progress, challenges, and opportunities in the face of COVID-19. Telehealth and Medicine Today, 6(1), 1–8. 10.30953/tmt.v6.238. [DOI] [Google Scholar]
- Portellano, J. A., Mateos, R., & Martínez Arias, M. R. (2022). CUMANES. In Cuestionario de Madurez Neuropsicológica escolar. Hogefre TEA Ediciones. [Google Scholar]
- Portellano, J. A., Mateos, R., Martínez Arias, M. R., & Sánchez-Sánchez, F. (2021). CUMANIN-2. In Cuestionario de Madurez Neuropsicológica Infantil-2. Hogefre TEA Ediciones. [Google Scholar]
- Postal, K. S., Bilder, R. M., Lanca, M., Aase, D. M., Barisa, M., Holland, A. A., et al. (2021). InterOrganizational practice committee guidance/recommendation for models of care during the novel coronavirus pandemic. The Clinical Neuropsychologist, 35 (1), 81–98. PubMed. 10.1080/13854046.2020.1801847. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Pritchard, A. E., Sweeney, K., Salorio, C. F., & Jacobson, L. A. (2020). Pediatric neuropsychological evaluation via telehealth: Novel models of care. The Clinical Neuropsychologist, 34(7–8), 1367–1379. 10.1080/13854046.2020.1806359. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Pulsifer, M. B., Grieco, J. A., Burstein, S. M., Parsons, M. W., Gardner, M. M., & Sherman, J. C. (2021). The development and implementation of teleneuropsychology in an academic lifespan neuropsychology center: Lessons learned from the COVID-19 pandemic. Journal of Clinical and Experimental Neuropsychology, 43(8), 1–12. 10.1080/13803395.2021.1963683. [DOI] [PubMed] [Google Scholar]
- Quiroga-Parra, D., Torrent-Sellens, J., & Murcia, C. (2017). Usos de las TIC en América Latina: Una caracterización. Ingeniare. Revista chilena de ingeniería, 25(2), 289–305. [Google Scholar]
- Ransom, D. M., Butt, S. M., DiVirgilio, E. K., Cederberg, C. D., Srnka, K. D., Hess, C. T.et al. (2020). Pediatric Teleneuropsychology: Feasibility and recommendations. Archives of Clinical Neuropsychology, 35(8), 1204–1214. 10.1093/arclin/acaa103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Roth, R. M., Isquith, P. K., & Gioia, G. A. (2005). BRIEF-A. behavior rating inventory of executive function-adult version. USA: Psychological Assessment Resources. [Google Scholar]
- Scott, T. M., Marton, K. M., & Madore, M. R. (2022). A detailed analysis of ethical considerations for three specific models of teleneuropsychology during and beyond the COVID-19 pandemic. The Clinical Neuropsychologist, 36(1), 24–44. 10.1080/13854046.2021.1889678. [DOI] [PubMed] [Google Scholar]
- Sherwood, A. R., & MacDonald, B. (2020). A Teleneuropsychology consultation service model for children with neurodevelopmental and acquired disorders residing in rural state regions. Archives of Clinical Neuropsychology, 35(8), 1196–1203. 10.1093/arclin/acaa099. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Singh, S., & Germine, L. (2021). Technology meets tradition: A hybrid model for implementing digital tools in neuropsychology. International Review of Psychiatry, 33(4), 382–393. 10.1080/09540261.2020.1835839. [DOI] [PubMed] [Google Scholar]
- Sociedad Mexicana de Psicología, S. M. P. (2010). Código Ético del Psicólogo (5th ed.). México: Trillas. [Google Scholar]
- Sozzi, M., Algeri, L., Corsano, M., Crivelli, D., Daga, M. A., Fumagalli, F.et al. (2020). Neuropsychology in the times of COVID-19. The role of the psychologist in taking charge of patients with alterations of cognitive functions. Frontiers in Neurology, 11, 1142. 10.3389/fneur.2020.573207. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stead, A., & Vinson, M. (2019). Cognitive assessment using face-to-face and videoconferencing methods. Nursing Older People, 31(5), 34–39. 10.7748/nop.2019.e1160. [DOI] [PubMed] [Google Scholar]
- Tailby, C., Collins, A. J., Vaughan, D. N., Abbott, D. F., O’Shea, M., Helmstaedter, C.et al. (2020). Teleneuropsychology in the time of COVID-19: The experience of the Australian epilepsy project. Seizure, 83, 89–97. PubMed 10.1016/j.seizure.2020.10.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Thibodaux, L. K., Breiger, D., Bledsoe, J., Sato, J., Hilsman, R., & Paolozzi, A. (2021). Teleneuropsychology: A model for clinical practice. Practice Innovations, 6(3), 189–198. 10.1037/pri0000150. [DOI] [Google Scholar]
- U.S. Department of Health & Human Services (2021). Notification of enforcement discretion for telehealth remote communications during the COVID-19 Nationwide public health emergency. https://www.hhs.gov/hipaa/for-professionals/special-topics/emergency-preparedness/notification-enforcement-discretion-telehealth/index.html. Www.HHS.Gov. [Google Scholar]
- Wadsworth, H. E., Dhima, K., Womack, K. B., Hart, J., Jr, Weiner, M. F., Hynan, L. S., et al. (2018). Validity of Teleneuropsychological assessment in older patients with cognitive disorders. Archives of Clinical Neuropsychology : The Official Journal of the National Academy of Neuropsychologists, 33(8), 1040–1045. PubMed. 10.1093/arclin/acx140. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wells, M. J., Dukarm, P., & Mills, A. (2021). Telehealth in rehabilitation psychology and neuropsychology. Physical Medicine and Rehabilitation Clinics, 32(2), 405–418. 10.1016/j.pmr.2020.12.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.