Abstract
Objective
To delineate a comprehensive curriculum for fellowship training in neuroinfectious diseases, we conducted a modified Delphi approach to reach consensus among 11 experts in the field.
Methods
The authors invited a diverse range of experts from the American Academy of Neurology Neuro-Infectious Diseases (AAN Neuro-ID) Section to participate in a consensus process using a modified Delphi technique.
Results
A comprehensive list of topics was generated with 101 initial items. Through 3 rounds of voting and discussion, a curriculum with 83 items reached consensus.
Conclusions
The modified Delphi technique provides an efficient and rigorous means to reach consensus on topics requiring expert opinion. The AAN Neuro-ID section provided the pool of diverse experts, the infrastructure, and the community through which to accomplish the consensus project successfully. This process could be applied to other subspecialties and sections at the AAN.
Infections of the nervous system have long been recognized as important in medicine, with tremendous public health and socioeconomic effects. Historically, diseases such as rabies, encephalitis lethargica, and poliomyelitis caused substantial neurologic sequelae. More recently, the global HIV/AIDS epidemic, outbreaks of West Nile and Zika virus diseases, and the development of postinfectious autoimmune syndromes have emerged as important contributors to neurologic injury and dysfunction.1 With the increasing role of specialized hospital-based care, acute neurologic infections constitute a major disease category requiring knowledge and expertise among neurohospitalists.2 Moreover, many individuals with neurologic infections are living longer, and there is increased interest in the long-term sequelae of chronic neuroinfectious disease in the United States and worldwide.3,4
Along with the shifting epidemiology and increased appreciation of the consequences of neurologic infections, there have been changes in training paradigms.5 While neuropathology comprised the major pillar of training in neuroinfectious diseases several decades ago, newer diagnostic and treatment modalities, recognition of autoimmune syndromes that necessitate distinction from neurologic infections, opportunities to interface with other specialists in the care of patients, and a changing regulatory environment with respect to medical training have resulted in additional educational needs. While a comprehensive list of neurology–infectious diseases (neuro-ID) fellowship training programs is lacking, an informal poll of neuro-ID section members combined with an Internet search found 10 neuro-ID fellowship training programs, each with 1–3 positions per year. The programs list a variety of clinical and research opportunities and include both dedicated neuro-ID and combined neuro-ID/neuroimmunology programs. The increasing interest in neuro-ID training is reflected in the American Academy of Neurology Neuro-Infectious Diseases (AAN Neuro-ID) Section membership, which increased from approximately 90 members in 2009 to more than 500 members as of 2019.5,6 Because neuro-ID does not have a formal accreditation status, the exact number of neuro-ID experts in the United States is unknown. As neuro-ID fellowship programs are relatively new, most senior neuro-ID experts completed postdoctoral research fellowships with informal clinical training while junior neuro-ID experts generally completed a formal fellowship in neuro-ID training.
Despite the importance of neurologic infections in human health, a formal subspecialty fellowship accreditation has not been designated and curricula for the training of fellows have not been agreed upon or standardized.5 The purpose of this study was to develop a competency-based neuroinfectious disease fellowship curriculum by leveraging the existing structure and expertise of the AAN Neuro-ID Section to form a consensus opinion of neurologists and neuroinfectious disease practitioners serving in diverse geographic and practice settings in the United States.
Methods
A modified Delphi method7,8 was used to establish group consensus on competencies that should be included in a comprehensive neuroinfectious disease fellowship curriculum. After establishing a comprehensive list of competencies, a select panel of 11 experts responded anonymously to questionnaires to rate their level of agreement for a competency being included in a neuroinfectious diseases curriculum. In the questionnaire, panelists could also provide comments on or revise the wording of competencies. Results, including comments, were summarized and presented to the panel in each round, with the goal of reducing the variation in responses on the next questionnaire and eventually arriving at a convergence of opinions among panelists. After each round, a revised questionnaire containing competencies on which consensus had not yet been reached was sent out to panelists, who were again asked to rate their level of agreement. This iterative process continued until the panel reached consensus on all competencies.
Expert panel
Through the AAN Neuro-ID section, the principal investigators (A.V., F.C.C., T.A.C.) extended invitations to neuroinfectious disease experts from diverse geographic and practice settings in the United States, and in different career stages. All participants on the expert panel had to be active clinicians with an interest in education and mentorship. A target of 10–12 panel members was selected based on literature review of the Delphi technique, balanced with feasibility and diverse representation. An initial meeting was held in person to lay the foundation for the Delphi process and to optimize buy-in for subsequent participation.
Questionnaire development
The principal investigators generated an initial broad set of competencies based on a prior AAN Neuro-ID section curriculum, review of textbooks and journal table of contents, and discussion. During the in-person meeting, participants had a chance to add items and to discuss the process. This initial questionnaire was piloted by 3 authors (A.V., F.C.C., T.A.C.), after which minor editorial and formatting changes were made to align with Accreditation Council for Graduate Medical Education milestone formatting. This edited questionnaire was submitted to the group electronically as the first round in the formal rating process. Panelists were asked to use a 4-point Likert scale (1: strongly disagree; 2: disagree; 3: agree; 4: strongly agree) to rate their level of agreement for a competency being included in the neuroinfectious diseases curriculum.
Data analysis and consensus building
For each round, descriptive statistics were applied including the mean score for each item and the proportion of responses at each level of agreement. In the first round, average scores clustered in the 3–4 range, with few items receiving ratings below 3. Therefore, consensus to exclude a competency from the curriculum was defined as an average rating of <3.0, whereas consensus to include a competency was defined as an average rating of >3.6. Competencies with an average rating between 3.0 and 3.6 were carried forward to the second round questionnaire. For the first round of voting only, items with comments that led to rewording for clarification were included for another vote regardless of the score. With each competency in the second round questionnaire, the item's ratings from the first round and panelists' anonymous comments were included. Based on feedback and discussion from the first round, 2 new competencies were added and several competencies were reworded in the second round questionnaire. In the third and final round, the remaining 11 competencies on which consensus had not been reached were discussed on a conference call until consensus was reached. For some items, this entailed combining items into a reworded competency.
Results
Eleven neuro-ID experts with a range of practice experience and from diverse geographic and practice settings were invited to complete the consensus curriculum process. All 11 panelists completed 2 rounds of the electronic survey, and 8 of 11 participated in the final round phone call, with the remaining 3 panelists confirming final consensus electronically. The round 1 questionnaire contained 101 individual items. Consensus for inclusion was reached on 63 items during round 1 (figure). The remaining 38 items were included in round 2. Based on qualitative feedback from round 1, several items were reworded and 2 new items were added. For round 2, all items were again displayed for reference but only the 40 items that had not reached consensus were available for rating. After round 2, 11 additional items reached consensus for inclusion, 18 items reached consensus for exclusion, and 11 items did not reach consensus. After the 3rd and final round during which 11 items were discussed, 1 item reached consensus for exclusion and the remaining 10 items reached consensus for inclusion through rewording and collapsing groups of items into single items (tables 1–3).
Figure. Consensus curriculum process map summarizes process for achieving consensus on neurology–infectious diseases curriculum.
Time in weeks refers to duration for each step in the process from preparation to execution. AAN Neuro-ID = American Academy of Neurology Neuro-Infectious Diseases.
Table 1.
Neurology–infectious diseases fellowship consensus curriculum, part 1: General clinical approach
Table 2.
Neurology–infectious diseases fellowship consensus curriculum, part 2: Syndromes and specific neuroinfectious diseases
Table 3.
Neurology–infectious diseases fellowship consensus curriculum, part 3: Syndromes and specific neuroimmune disorders
Multiple competencies associated with clinical care enjoyed strong support in our Delphi process. With respect to the clinical and diagnostic evaluation, we found that there was particularly strong support for competency in elements of the neuroinfectious disease–focused history, in the use and interpretation of neuroimaging and CSF in diagnosis, and in the therapeutic and prophylactic use of antimicrobial agents. Strong support was also noted for a team-based approach to clinical care, as exemplified by competencies in the appropriate use of consultative services. In addition, competencies in systems-based issues such as educating others on clinical reasoning and exhibiting familiarity with hospital, local, and state policies on infection control and reporting of infections, and referrals to appropriate infectious and autoimmune resources and support groups, achieved strong support. Strong support was also given to knowledge-based competencies addressing neuroinfectious and autoimmune disorders and syndromes. On the other hand, competencies associated with the routine use of specialized neurologic examination scales (e.g., HIV neuropathy scale, Memorial Sloan-Kettering scale) were not rated highly, exhibited lower scores in round 2 than round 1, and were excluded. Several competencies associated with symptomatic management of patients, including those addressing issues of fatigue, urinary problems, and sexual issues, were also excluded as they were not believed to be within the direct purview of neuroinfectious disease fellowship training. Finally, competencies associated with several diagnostic tests also did not achieve consensus, though for differing reasons: evoked potentials were believed to add relatively little information in light of currently available diagnostic tests, while optical coherence tomography was believed to be promising though not yet proven to add substantially to the diagnostic process.
Discussion
This study demonstrates the utility of the Delphi method in establishing a consensus curriculum for a neurologic subspecialty fellowship. Leveraging the community and expertise of the AAN Neuro-ID section, a diverse set of highly invested panelists were invited to participate in the consensus project and all 11 experts completed the full process. Despite varied backgrounds, geographic locations, and practice settings, this panel of experts successfully reached consensus on a variety of fellowship competencies spanning knowledge, evaluation, and treatment of neuroinfectious diseases. Overall, the consensus neuroinfectious diseases fellowship curriculum includes competencies that address clinical care, team-based approaches, systems-based competencies, knowledge domains, and dissemination of knowledge. This competency-based curriculum can provide a foundation for training fellows across the country and potentially globally. Moreover, the approach used may serve as a framework for development of educational curricula by other sections of the AAN.
The Delphi process has been used to address a variety of issues in neurologic education, including the delineation of core neurologic examination items for neurology clerkships, timing of milestone competency acquisition and the nature of observable practice activities in neurology residency, optimization of neuroscience grand rounds, and development of a neurology health advocacy curriculum for trainees.9–14 The process is most useful when there is little empirical evidence to inform considerations,15 and is thus particularly suitable for the development of a neuroinfectious diseases fellowship curriculum where no formal accreditation exists for the subspecialty. One of the most important features of the Delphi process is the identification of a range of experts with sufficient commitment to see the process through,8,16,17 which in our case was facilitated by the structure of the AAN Neuro-ID section. The initial meeting was held in person during an annual AAN meeting, both to garner investment among panelists and to generate a comprehensive list of items efficiently. Indeed, all 11 experts were highly invested throughout and participated in the process through completion. In a traditional Delphi process, the use of anonymous voting is believed to protect against the influence of more senior/established experts causing bias in the results. We modified this approach in the final round by convening a conference call so that we could more efficiently come to an understanding about the remaining concerns that precluded consensus. After 2 anonymous rounds, we believed that the lack of anonymity in the final round was outweighed by the efficiency and clarity that could be achieved with a live discussion. The entire process took 11 weeks (figure). Overall, we envision that this approach could be utilized by other AAN sections to effectively and efficiently develop subspecialty-specific curricula directed toward fellows, residents, or medical students, perhaps with integration of the AAN Synapse online platform in order to facilitate communication and eliminate the need for an initial in-person meeting.
Several limitations should be noted. While a diverse array of experts from different regions and in differing practice settings were included, there remains a possibility of bias related to participant background.17 International experts were not included. While the curriculum set forth an outline of competencies in neuroinfectious diseases, resources to develop mastery of these competencies—digital, print, or otherwise—were not addressed in this initial Delphi process. We are currently planning a follow-up modified Delphi approach, again through the AAN Neuro-ID section, to develop a listing of resources that complements the curriculum. Ideally, the comprehensive curriculum included here could serve as a foundation for neuro-ID topics essential for any general neurologist, but the current process focused on expert consensus rather than distilling the core items appropriate for residency training.
As with any curriculum, assessment methods are essential for evaluating both the curriculum and the trainee's attainment of competencies. We plan to poll fellowship directors regarding the utility and feasibility of the curriculum in practice. Obtaining data in exit interviews on how many of the curricular competencies trainees achieved during their fellowship would also be valuable. The current study did not formally address tools to assess trainees, but we envision a combination of methods for evaluating competency in this curriculum: clinical proctoring and online evaluation of competencies, maintenance and review of a case log, journal club discussion of key references for topics in the curriculum, trainee-authored abstracts and manuscripts covering curricular items, and virtual cases with knowledge assessments.
Of note, neuroinfectious diseases pose a unique challenge to curriculum development as they are ever-changing, with new syndromes and disorders emerging rapidly and on a regular basis.1 As a result, mechanisms to append or update curricula are vital, and could involve regularly scheduled updates to the Delphi process or mechanisms to provide comments or updates in real time on an electronic version of the curriculum. While the development of some curricula can be informed by the presence of questions on standardized tests such as subspecialty board examinations,18 this was not possible for neuroinfectious diseases, where no such subspecialty-specific certification examination exists. Rather, we envision that our curriculum may guide future efforts to ensure consistency of training in neuroinfectious disease.
The Delphi process is a powerful and efficient tool to reach consensus on neurology subspecialty competencies. By adapting the process to the specific context of neuroinfectious diseases and leveraging the AAN Neuro-ID section, we were able to reach consensus on a comprehensive curriculum for training in neuroinfectious diseases. This process could serve as a template for creating or modifying other subspecialty curricula as well as resources to achieve those competencies.
Acknowledgment
Cheryl Jay is deceased.
Glossary
- AAN Neuro-ID
American Academy of Neurology Neuro-Infectious Diseases
- neuro-ID
neurology–infectious diseases
Appendix. Authors
Study funding
No targeted funding reported.
Disclosure
The authors report no disclosures relevant to the manuscript. Go to Neurology.org/N for full disclosures.
References
- 1.Nath A. Grand challenges in neuroinfectious diseases. Front Neurol 2017;8:480. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Tan K, Patel S, Gandhi N, Chow F, Rumbaugh J, Nath A. Burden of neuroinfectious diseases on the neurology service in a tertiary care center. Neurology 2008;71:1160–1166. [DOI] [PubMed] [Google Scholar]
- 3.Levin SN, Lyons JL. Infections of the nervous system. Am J Med 2018;131:25–32. [DOI] [PubMed] [Google Scholar]
- 4.Wright EJ, Thakur KT, Bearden D, Birbeck GL. Global developments in HIV neurology. Handb Clin Neurol 2018;152:265–287. [DOI] [PubMed] [Google Scholar]
- 5.Millichap JJ, Epstein LG. Emerging subspecialties in neurology: neuroinfectious diseases. Neurology 2009;73:e14–e15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.AAN Synapse. Available at: synapse.AAN.com/communities. Accessed April 17, 2019.
- 7.Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs 2000;32:1008–1015. [PubMed] [Google Scholar]
- 8.de Villiers MR, de Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach 2005;27:639–643. [DOI] [PubMed] [Google Scholar]
- 9.Abuzinadah AR, Cooke L. Neurology health advocacy curriculum: needs assessment, curricular content and underlying components. Can J Neurol Sci 2017;44:170–176. [DOI] [PubMed] [Google Scholar]
- 10.Jones LK, Dimberg EL, Boes CJ, et al. Milestone-compatible neurology resident assessments: a role for observable practice activities. Neurology 2015;84:2279–2283. [DOI] [PubMed] [Google Scholar]
- 11.Jones LK, Eggers SDZ, Capobianco DJ, Boes CJ. Timing of milestone competency acquisition in neurology residency: what by when? Neurology 2018;91:748–754. [DOI] [PubMed] [Google Scholar]
- 12.Liu CH, Hsu LL, Hsiao CT, et al. Core neurological examination items for neurology clerks: a modified Delphi study with a grass-roots approach. PLoS One 2018;13:e0197463. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Moore FG, Chalk C. The essential neurologic examination: what should medical students be taught? Neurology 2009;72:2020–2023. [DOI] [PubMed] [Google Scholar]
- 14.Rigby H, Schofield S, Mann K, Benstead T. Education Research: an exploration of case-based learning in neuroscience grand rounds using the Delphi technique. Neurology 2012;79:e19–e26. [DOI] [PubMed] [Google Scholar]
- 15.Powell C. The Delphi technique: myths and realities. J Adv Nurs 2003;41:376–382. [DOI] [PubMed] [Google Scholar]
- 16.Green B, Jones M, Hughes D, Williams A. Applying the Delphi technique in a study of GPs' information requirements. Health Soc Care Community 1999;7:198–205. [DOI] [PubMed] [Google Scholar]
- 17.Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud 2001;38:195–200. [DOI] [PubMed] [Google Scholar]
- 18.Salas RME, Strowd RE, Ali I, et al. Incorporating sleep medicine content into medical school through neuroscience core curricula. Neurology 2018;91:597–610. [DOI] [PubMed] [Google Scholar]