Global uncertainty in the diagnosis of neurological complications of SARS-CoV-2 infection by both neurologists and non-neurologists: An international inter-observer variability study

Abstract

Introduction

Uniform case definitions are required to ensure harmonised reporting of neurological syndromes associated with SARS-CoV-2. Moreover, it is unclear how clinicians perceive the relative importance of SARS-CoV-2 in neurological syndromes, which risks under- or over-reporting.

Methods

We invited clinicians through global networks, including the World Federation of Neurology, to assess ten anonymised vignettes of SARS-CoV-2 neurological syndromes. Using standardised case definitions, clinicians assigned a diagnosis and ranked association with SARS-CoV-2. We compared diagnostic accuracy and assigned association ranks between different settings and specialties and calculated inter-rater agreement for case definitions as “poor” (κ ≤ 0.4), “moderate” or “good” (κ > 0.6).

Results

1265 diagnoses were assigned by 146 participants from 45 countries on six continents. The highest correct proportion were cerebral venous sinus thrombosis (CVST, 95.8%), Guillain-Barré syndrome (GBS, 92.4%) and headache (91.6%) and the lowest encephalitis (72.8%), psychosis (53.8%) and encephalopathy (43.2%). Diagnostic accuracy was similar between neurologists and non-neurologists (median score 8 vs. 7/10, p = 0.1). Good inter-rater agreement was observed for five diagnoses: cranial neuropathy, headache, myelitis, CVST, and GBS and poor agreement for encephalopathy. In 13% of vignettes, clinicians incorrectly assigned lowest association ranks, regardless of setting and specialty.

Conclusion

The case definitions can help with reporting of neurological complications of SARS-CoV-2, also in settings with few neurologists. However, encephalopathy, encephalitis, and psychosis were often misdiagnosed, and clinicians underestimated the association with SARS-CoV-2. Future work should refine the case definitions and provide training if global reporting of neurological syndromes associated with SARS-CoV-2 is to be robust.

Graphical abstract

1. Introduction

Up to a third of all adults infected with SARS-CoV-2 develop neurological or psychiatric symptoms within the first six months of infection [1] and half of hospitalised patients with such manifestations have poor prognosis [2]. Yet, the global study of neurological complications of SARS-CoV-2 infection is challenging, due to the relative rarity of individual disease entities and complexities of making neurological diagnoses, which is amplified by limited access to neurological expertise, workforce and diagnostic investigations [3]. The Global COVID-19 Neuro-Research Coalition was set up to tackle some of these challenges, bringing together neurologists and allied clinicians investigating COVID-19 across the globe [4].

Through this Coalition, an inconsistency in case definitions used in published studies was identified as a key challenge limiting multi-centre recruitment and data pooling in COVID-19 neurology research [3]. Several case definitions have been proposed [5], but none have been externally validated, and it is not known whether diagnostic accuracy varies between different neurological complications. It is also unclear if the perceived degree of association between a given neurological diagnosis and SARS-CoV-2 infection varies between neurologists and allied clinicians as well as between different practice settings and levels of experience. If present, such variation could impact the likelihood of reporting and recruiting patients to research studies, hampering recognition of neurological manifestations of SARS-CoV-2 infection and, consequently, timely and appropriate management for patients.

To investigate these uncertainties, in collaboration with the World Federation of Neurology and the Global COVID-19 Neuro-Research Coalition, we surveyed neurologists and other healthcare professionals providing care for patients with COVID-19 worldwide. The aim was firstly, to analyse the utility of the developed case definitions of neurological complications of SARS-CoV-2 infection in supporting clinicians in the diagnostic process and secondly, to assess whether clinicians' experience and practice setting affect both the diagnostic accuracy and the perceived causality of SARS-CoV-2 in the neurological complications.

2. Methods

We created an online survey portal, open to any healthcare professional worldwide, through which respondents assessed ten anonymised vignettes, describing real-life scenarios of acute neurological complications of SARS-CoV-2 infection. These were identified by an expert panel of members of the Global COVID-19 Neuro-Research Coalition to reflect the broad spectrum of the neurological complications of SARS-CoV-2 infection in varying clinical settings and to combine syndromes which may present similarly but require different treatment approaches. Three vignettes came from the UK, two each from the USA and India and one each from Italy, Spain and Myanmar. The ten syndromes comprised: headache, ischaemic stroke, cerebral venous-sinus thrombosis (CVST), encephalopathy, encephalitis, cranial neuropathy, Guillain-Barré syndrome (GBS), myelitis, seizures and psychosis (supplementary material). Five of the ten cases had been published separately previously [[6], [7], [8], [9], [10]]. All vignettes were edited for consistency of presentation style and word count. The diagnosis made by the treating clinicians was used as the gold standard following verification by the Global COVID-19 Neuro-Research Coalition expert panel. At the time of the development of this study, internationally established diagnostic criteria for long-COVID were not agreed, therefore we focused on acute neurological complications associated with SARS-CoV-2 infection.

For each of the ten scenarios, participants answered three multiple-choice questions. Firstly, the diagnosis had to be selected from a list of diagnostic labels. To support this, we provided participants with case definitions (supplementary material). These definitions were first created to set up neurological registries at the beginning of the COVID-19 pandemic and subsequently became used in observational studies [11]. Additional terms were added by the members of the Global COVID-19 Neuro-Research Coalition from published definitions [5] and the World Health Organization (WHO) COVID-19 case report form [12]. Secondly, participants were asked to rate their confidence in the diagnosis they had assigned on a five-point Likert scale. Lastly, participants ranked the association of the neurological complication with SARS-CoV-2 as either “confirmed”, “probable”, “possible” or “unlikely”, based on the provided definitions [5,13]. The correct association rank for each scenario was determined prospectively by the expert panel of members of the Coalition in line with the provided definitions.

Prior to the assessment of the ten cases, participants were presented with an example training vignette and asked to provide data on their country and practice setting, profession, years of professional experience and their exposure to both patients with COVID-19 and patients with neurological conditions. The survey portal was accessed in English and was advertised through a series of international channels: the World Federation of Neurology (WFN) newsletter, electronic correspondence with the WFN-affiliated national neurological societies [14], at the World Congress of Neurology and via direct contacts of the members of the Global COVID-19 Neuro-Research Coalition [15]. The completion of the survey took 30 to 40 min. Correct answers were not revealed to participants during the survey but could be requested as individual feedback after study closure.

Data were collected, stored, processed and analysed using RedCap, Microsoft Excel and SPSS (IBM, version 28). Distribution of variables was examined using histograms, Q-Q plots and Shapiro-Wilk normality test. Non-parametric tests were employed for comparisons of continuous variables. Nominal variables were compared using Chi-square or Fisher's exact test for cell counts <5. Fleiss-kappa was used to examine inter-rater agreement for diagnostic and association categories. Agreement was interpreted as “poor” for κ ≤0.40, “good” for κ >0.60 and “moderate” between these cut-offs. Spearman coefficient was calculated to assess correlation between clinical experience and the proportion of correct answers. P values <0.05 were considered statistically significant. All vignettes were fully anonymised and no ethical approval was required for the study.

3. Results

3.1. Participants

In six months, we received entries from 191 individuals, of whom 146 (76.4%) assessed at least one case scenario and 119 (62.3%) assessed all ten. Individuals who did not assess any scenarios were excluded from further analysis.

The 146 participants came from 45 countries on six continents, with the highest number of participants from the Philippines (21; 14.4%), Türkiye (14; 9.6%) and Croatia (14; 9.6%) (Fig. 1 ). Of the participants, 133 (91.1%) were doctors; of these 111 (83.5%) were neurologists (including two paediatric neurologists). Doctors from non-neurology backgrounds included general internal physicians (10; 6.9%), other hospital-based specialists (9; 6.2%) and doctors working in other settings (3; 2.1%). The thirteen allied professionals (8.9%) comprised clinical officers (10; 6.9%) and a nurse specialist, a pharmacist and a clinical psychologist. More participants were based in referral hospitals (106; 73.3%) than in general hospitals (30, 19.9%). Five worked in primary care and five in research settings (3.4% each).

Fig. 1.

Countries of origin (in blue) of the 146 survey respondents.*

*Created using MapChart [20]. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

The median (IQR; range) number of years in practice was 13 (7–21; 1–47). Over a half of participants (76, 52.1%) had >10 years' experience in treating patients with neurological conditions and most (131, 89.7%) reported seeing such patients daily, but these proportions were higher among neurologists than non-neurologists (p < 0.001; Table 1 ). Most participants (n = 109; 74.7%) had at least a month experience looking after patients with COVID-19, but neurologists looked after patients with COVID-19 more rarely than other survey respondents (p = 0.008).

Table 1.

Experience with care of patients with neurological conditions and patients with COVID-19 among the 111 neurologists and 35 non-neurologists who responded to the survey.

Clinical experience domain		Neurologists (n = 111)	Non-neurologists (n = 35)	Statistical comparison (Fisher's exact test)
Duration of experience with predominantly looking after patients with neurological conditions	>10 years	61 (55%)	15 (42.9%)	p < 0.001
	Between 5 and 10 years	17 (15.3%)	7 (20%)
	Between 1 year and 5 years	33 (29.7%)	3 (8.6%)
	<1 year	0	8 (22.9%)
	None	0	2 (5.7%)
Frequency of clinical encounters with patients with neurological conditions during routine work	Multiple patients daily	102 (91.9%)	22 (62.9%)	p < 0.001
	One or two patients daily	5 (4.5%)	2 (5.7%)
	Weekly	3 (2.7%)	4 (11.4%)
	Monthly	1 (0.9%)	2 (5.7%)
	Less than monthly	0	5 (14.3%)
Duration of experience with predominantly looking after patients with COVID-19, throughout the COVID-19 pandemic	More than a year	50 (45%)	15 (42.9%)	p = 0.72
	Between 6 months and a year	11 (9.9%)	6 (17.1%)
	Between 1 month and 6 months	22 (19.8%)	5 (14.3%)
	<1 month	12 (10.8%)	5 (14.3%)
	None	16 (14.4%)	4 (11.4%)
Frequency of clinical encounters with patients with COVID-19, at the peak of the COVID-19 pandemic	Multiple patients daily	23 (20.7%)	17 (48.6%)	p = 0.008
	One or two patients daily	18 (16.2%)	2 (5.7%)
	Weekly	35 (31.5%)	4 (11.4%)
	Monthly	14 (12.6%)	5 (14.3%)
	Less than monthly	21 (18.9%)	7 (20%)

3.2. Diagnostic accuracy

A total of 1265 diagnoses were recorded by the survey participants across the ten scenarios (Table 2 ). Of these, 965 were correct (76.3% [95% CI 73.9% –78.6%]). The proportion of the correct diagnoses was higher for conditions where respondents reported greater diagnostic confidence (Table 2). CVST was diagnosed correctly by 95.8% respondents, followed by GBS (92.4%) and headache (91.6%). The lowest proportion of correct answers was recorded for encephalopathy (43.2%), psychosis (53.8%) and encephalitis (72.8%). Encephalitis was most often misdiagnosed as primary epileptic seizures (5.6%), encephalopathy (4.8%) or ischaemic stroke (4.8%; Fig. 2 ). Psychosis was most frequently misdiagnosed as encephalopathy (28.6%) or encephalitis (5%). Encephalopathy was misdiagnosed as ischaemic stroke (28.8%), psychiatric dysexecutive syndrome (8.9%) or encephalitis (5.5%).

Table 2.

Confidence in diagnosis, proportion of correct responses and strength of agreement among 146 clinicians worldwide assigning diagnoses to ten real-life scenarios of acute neurological complications of COVID-19.

Neurological Syndrome⁎	Number of respondents (n)	Mean confidence in diagnosis (0−10)⁎⁎	Proportion of correct diagnosis (%)	95% CI (%)	Strength of agreement (Fleiss Kappa)	95% CI (Fleiss Kappa)
Cerebral venous thrombosis	120	8.77	95.8	92.3–99.4	0.676	0.670–0.682
Guillain-Barré syndrome	119	8.89	92.4	87.7–97.2	0.622	0.616–0.628
Headache	119	7.67	91.6	86.6–96.6	0.702	0.696–0.708
Myelitis	127	8.47	81.9	75.2–88.6	0.688	0.682–0.694
Seizure	123	7.34	80.5	73.5–87.5	0.599	0.593–0.605
Ischaemic Stroke	130	7.42	79.2	72.3–86.2	0.476	0.470–0.482
Cranial neuropathy	137	7.15	78.1	71.2–85.0	0.710	0.704–0.716
Encephalitis	125	7.32	72.8	65.0–80.6	0.422	0.416–0.428
Psychosis	119	7.14	53.8	44.8–62.7	0.409	0.403–0.415
Encephalopathy	146	6.80	43.2	35.1–51.2	0.257	0.251–0.263

^⁎Cases are ranked by the proportion of respondents assigning the correct diagnoses, such that the cases with the lowest numbers of correct answers are at the bottom. The overall agreement across all diagnoses was moderate (Fleiss Kappa = 0.443, 95% CI 0.441–0.445).

^⁎⁎Mean confidence in diagnosis was derived for each scenario by transforming ordinal variables from Likert scale into a continuous (0 to 10) scale, as described previously [19]. Score 10 would indicate that all respondents rated their confidence as “very high” whereas score 0 would indicate that all respondents rated their confidence as very low.

Fig. 2.

Diagnostic labels assigned to each clinical scenario.

Each bar represents one of the ten clinical scenarios. The correct diagnosis is given at the bottom of each bar (x axis). The y axis shows the proportion of the diagnostic labels selected for each scenario by the study respondents. Each diagnostic label is represented by a different colour, including the correct diagnostic labels for the ten clinical scenarios and other diagnostic labels. The diagnostic labels selected by four or fewer respondents for each clinical scenario were combined into an “other” category.

There was no significant difference in the number of correct diagnoses between neurologists (n = 111; median score 8/10 [IQR 6–9]) and non-neurologists (n = 35, median score 7/10 [IQR 5–9]; p = 0.11). Healthcare professionals based at tertiary centres or in research settings (n = 111, median score 8/10 [IQR 6–9]) performed better than those based elsewhere (n = 35, median score 7/10 [IQR 6–8]; p = 0.024). The number of years spent in neurology did not correlate with the proportion of correct diagnoses (r _s = 0.114, p = 0.87). The frequency of clinical encounters with patients with neurological conditions had a marginal correlation with correct answers (r _s = 0.160; p = 0.054).

The overall agreement across all case definitions was moderate (κ = 0.443 [95% CI 0.441–0.445]; Table 2). Good strength of agreement was observed for five syndromes: cranial neuropathy (κ = 0.71), headache (κ = 0.70), myelitis (κ = 0.688), CVST (κ = 0.68) and GBS (κ = 0.62). Moderate strength of agreement was found for ischaemic stroke (κ = 0.476), encephalitis (κ = 0.42), psychosis (κ = 0.41) and poor for encephalopathy (κ = 0.25).

3.3. Association with COVID-19

A total of 1265 association ranks were recorded by the survey respondents across the ten scenarios, of which 965 (59.1% [95%CI 56.4–61.8]) were correct when referenced with the study definitions. 78.4% (95% CI 71.2–85.6) respondents assigned the “confirmed” association rank correctly, but this proportion was lower for “probable” (56.8% [95% CI 53.3–60.4]) and “possible” (57.3% [95% CI 52.3–62.2]) association ranks. Inter-rater agreement was poor across the categories (κ = 0.129 [95%CI 0.126–0.133]), except for the “confirmed” rank (κ = 0.3 [95% CI 0.293–0.305]).

There was no difference in the number of correctly assigned association ranks between neurologists (n = 111; median 5/10 [IQR 3–8]) and non-neurologists (n = 35; median 6/10 [IQR 4–8]; p = 0.66) and no difference between the practice settings (median 6/10 [IQR 4–8] for respondents from tertiary or research settings [n = 111] and median 5/10 [IQR 2–8] for respondents from primary and secondary care [n = 35]; p = 0.15). There was also no correlation between the proportion of correctly assigned association ranks and respondents' clinical exposure to patients with COVID-19, both in terms of months of experience (r _s = −0.132; p = 0.11) and the frequency of clinical encounters (r _s = −0.056; p = 0.50).

Overall, respondents assigned 13% more cases to the lower two association ranks than would have been assigned correctly (p < 0.001; Fig. 3 ). There was no difference in the distribution of the assigned association ranks between neurologists and non-neurologists (p = 0.86), nor between respondents based in a tertiary or research setting and those based elsewhere (p = 0.46).

Fig. 3.

Distribution of association ranks assigned by all survey respondents, and respondents stratified by specialty and by setting.

Compared to the distribution of correct answers, survey respondents assigned an additional 13% of cases to the lower association ranks (p < 0.001). There was no difference in the association ranks assigned by the neurologists and non-neurologists (p = 0.86) and no difference between respondents based in tertiary or research settings and those based elsewhere (p = 0.46).

4. Discussion

In this online survey, 146 clinicians from 45 countries on six continents assessed ten real-life vignettes of neurological complications of SARS-CoV-2 infection, assigned a diagnosis and ranked the likelihood of association with SARS-Cov-2. In the presence of the standardised case definitions, the correct diagnosis was assigned by >75% of respondents for seven of the ten vignettes, with comparable diagnostic accuracy between neurologists and non-neurologists. Six of the ten case definitions showed good inter-rater reliability, validating their use in future studies and in settings with few neurologists. Conversely, syndromes presenting with altered mental status, such as psychosis, encephalopathy and encephalitis, were frequently misdiagnosed. Respondents also underestimated association between neurological syndromes and SARS-CoV-2 infection in 13% of cases, regardless of their specialty or practice setting. These two areas should be addressed in future work to ensure that global reporting of neurological syndromes and their association with SARS-CoV-2 is robust.

The ten syndromes examined in this study represent clinically significant neurological conditions that have been well-described in association with SARS-CoV-2 infection [16,17]. Although some of these syndromes are rare, we selected them based on the availability of outcome-modifying treatments, which makes establishing the correct diagnosis of paramount importance. Our aim was to assemble syndromes which may present similarly but require different treatment approaches. Examples are: paraparesis secondary to myelitis or GBS; confusion resulting from encephalopathy, encephalitis or psychosis; or limb weakness secondary to CVST or ischaemic stroke. With provision of the standardised case definitions, we showed comparable diagnostic accuracy between neurologists and non-neurologists, despite significant differences in clinical experience. This supports the use of these definitions outside specialist neurology settings.

Good diagnostic accuracy and inter-rater agreement were recorded even for scenarios perceived as more difficult by the respondents, such as cranial neuropathy. The exception to this were the conditions primarily presenting with altered mental status: encephalitis, psychosis and encephalopathy. Despite explicitly normal brain imaging, a large proportion of respondents labelled encephalopathy as ischaemic stroke, likely because the scenario described speech difficulty. Apart from ischaemic stroke, encephalopathy, encephalitis and psychosis were frequently misdiagnosed for each other, even though the scenarios focused on the core features enclosed in the case definitions, such as signs of brain parenchymal involvement and evidence of cerebrospinal pleocytosis in the encephalitis vignette. We suspect that this reflects a global uncertainty in distinguishing between these conditions, which, in clinical practice, is also challenging due to complex and overlapping clinical presentations. However, with encephalopathy among the most common neurological manifestation of COVID-19 [2,16], and requiring primarily supportive rather than immunosuppressive treatment [18], further work should focus on designing tools to help clinicians differentiate between these syndromes.

Despite the multitude of reports on COVID-19-related neurological syndromes, respondents underestimated the association between SARS-CoV-2 infection and neurological syndromes in over 10% of cases. Ranking association as “confirmed”, “probable”, “possible”, or “unlikely” required consideration of the temporal relationship, evidence of SARS-CoV-2 infection and alternative aetiologies and risk factors [5,13]. According to the provided case definitions, presence of alternative aetiologies stipulates no higher association rank than “possible”, but what constitutes a plausible alternative aetiology is subject to the assessor's clinical judgement. This may explain why some vignettes were assigned a “probable” rather than “possible” association rank. Clinicians should therefore be encouraged to report cases even if they perceive association as possible, as underestimation of causality may affect clinical management and recruitment of patients to registries and prospective studies, limiting recognition of the full spectrum of neurological complications of SARS-CoV-2 infection.

We aimed to limit bias by not providing financial incentives and by distributing through broad networks. Nevertheless, it is possible that clinicians more confident in clinical neurology might have been more likely to participate, which may result in a better diagnostic accuracy. Despite extensive distribution efforts, participants from primary or secondary care or from settings where neurological expertise is limited were somewhat underrepresented. The online portal was offered only in English and this limits the generalisability of the results and may have affected the accuracy of responses in participants less comfortable with English. However, few other surveys have achieved such a geographical representation, and none have investigated the diagnostic or causality assessment process of clinicians in relation to neurological syndromes associated with SARS-CoV-2 infection. Our results therefore provide foundations for further work into supporting healthcare professionals worldwide in diagnosis and reporting, and similar effort should be undertaken for other infections with neurological complications. That diagnostic accuracy was similar between neurologists and non-neurologists in the presence of the case definitions is encouraging for their application in settings with few or no neurologists. Nevertheless, future work is required to improve the diagnosis of patients presenting with altered mental status in association with SARS-CoV-2 infection to facilitate the distinction between encephalopathy, encephalitis, and psychosis, in part through refining of the case definitions and clinical training. Revised clinical definitions could use a two-step case categorisation; by first selecting the presenting symptom (such as altered mental status) and then the final, post-investigation diagnosis (such as encephalitis). This would enhance the clarity by differentiating symptoms from neurological conditions, while simplifying the diagnostic process. Such revised definitions will be validated in future follow-up studies.

In conclusion, this multicentre study found that, when supported with the case definitions, both neurologists and non-neurologists achieved good diagnostic accuracy and inter-observer agreement for many neurological syndromes associated with SARS-CoV-2. The exception was syndromes presenting with altered mental status, such as encephalitis, encephalopathy and psychosis. In addition, clinicians may underestimate the association of neurological syndromes with SARS-CoV-2 infection. Further work should focus on addressing these two areas and evaluation of future clinical tools retrospectively and prospectively in clinical practice. Global consensus agreement between the diagnosis and causality assessment of neurological complications of SARS-CoV-2 are critical to support clinical diagnosis, registry reporting and study recruitment, and have implications for other infections causing neurological complications.

Disclaimer

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the World Health Organization.

Funding

AAT, GKW and GW are funded by the UK National Institute for Health Research (NIHR) as NIHR academic clinical fellows and MAE and TP as NIHR academic clinical lecturers. EW was supported by the School of Medicine, Technical University of Munich, through Grant Number H.40001.1.7–08 for the coordination of the Global COVID-19 Neuro Research Coalition. AÖ is the current president of the Global Migraine and Pain Society and Mersin Alzheimer Society. KTT is funded by the National Institute of Health, Centers for Disease Control and Prevention. TS is supported by The Pandemic Institute, and The National Institute of Health and Care Research (NIHR) Health Protection Research Unit in Emerging and Zoonotic Infections. B.D.M. is supported to con- duct COVID-19 neuroscience research by the UKRI/ MRC (MR/V03605X/1). B.D.M. is also supported for additional neurological inflammation research due to viral infection by grants from the NIHR (award CO-CIN-01), the Medical Research Council (MC_PC_19059) and by the NIHR Health Protection Research Unit (HPRU) in Emerging and Zoonotic Infections at University of Liver- pool in partnership with Public Health England (PHE), in collaboration with Liverpool School of Tropical Medi- cine and the University of Oxford (award 200907), NIHR HPRU in Respiratory Infections at Imperial College Lon- don with PHE (award 200927), the MRC/UKRI (MR/ V007181/1), MRC (MR/T028750/1), and Wellcome (ISSF201902/3).

Declaration of Competing Interest

KTT is an external consultant at the World Health Organization and the Clinical Immunization Safety Assessment Committee, CDC. BDM is an external consultant at the World Health Organization.

Appendix A. Supplementary data

Supplementary material 1

Supplementary material 2