Accuracy of Clinician Suspicion of Lyme Disease in the Emergency Department

In our multicenter prospective cohort of children being evaluated for Lyme disease, clinician suspicion provided only minimal discriminative ability, highlighting the need for confirmatory testing.

Abstract

BACKGROUND:

To make initial management decisions, clinicians must estimate the probability of Lyme disease before diagnostic test results are available. Our objective was to examine the accuracy of clinician suspicion for Lyme disease in children undergoing evaluation for Lyme disease.

METHODS:

We assembled a prospective cohort of children aged 1 to 21 years who were evaluated for Lyme disease at 1 of the 5 participating emergency departments. Treating physicians were asked to estimate the probability of Lyme disease (on a 10-point scale). We defined a Lyme disease case as a patient with an erythema migrans lesion or positive 2-tiered serology results in a patient with compatible symptoms. We calculated the area under the curve for the receiver operating curve as a measure of the ability of clinician suspicion to diagnose Lyme disease.

RESULTS:

We enrolled 1021 children with a median age of 9 years (interquartile range, 5–13 years). Of these, 238 (23%) had Lyme disease. Clinician suspicion had a minimal ability to discriminate between children with and without Lyme disease: area under the curve, 0.75 (95% confidence interval, 0.71–0.79). Of the 554 children who the treating clinicians thought were unlikely to have Lyme disease (score 1–3), 65 (12%) had Lyme disease, and of the 127 children who the treating clinicians thought were very likely to have Lyme disease (score 8–10), 39 (31%) did not have Lyme disease.

CONCLUSIONS:

Because clinician suspicion had only minimal accuracy for the diagnosis of Lyme disease, laboratory confirmation is required to avoid both under- and overdiagnosis.

What’s Known on This Subject:

Treating clinicians evaluating children for potential Lyme disease must make initial management decisions before results of Lyme disease tests are available. The accuracy of clinician suspicion for the diagnosis of Lyme disease has not been rigorously evaluated.

What This Study Adds:

Clinician suspicion had minimal accuracy for the diagnosis of Lyme disease. Although initial treatment may be guided by clinician suspicion, confirmatory 2-tiered Lyme disease serology should be used to avoid either under- or overdiagnosis of Lyme disease.

In Lyme disease–endemic areas, children frequently develop Borrelia infections after an unrecognized Ixodes scapularis tick bite.¹ Common clinical manifestations include an erythema migrans (EM) lesion, facial palsy, meningitis, and arthritis.² However, Lyme “mimics” (ie, patients with clinical symptoms compatible with possible Lyme disease but who ultimately were found to have alternate diagnoses) cause initial diagnostic and therapeutic uncertainty. Because Lyme disease serology takes several days to return results, clinicians must make initial patient management decisions before diagnostic test results are available. Although children with Lyme disease require prompt initiation of appropriate antibiotics, diagnosis of Lyme mimics may require invasive procedures, such as lumbar puncture or arthrocentesis for diagnosis, as well as alternate treatments.

Clinicians use clinical prediction rules to combine available demographic, clinical, and laboratory factors to estimate the probability of an outcome and assist clinical decision-making.³ Clinicians also use prediction rules to estimate the probability of Lyme disease for children presenting with facial palsy,^4,5 meningitis,^6–8 and arthritis.^9–11 With these models, we can identify children at the lowest and the highest risk of Lyme disease and can guide initial clinical decisions while awaiting the results of diagnostic testing. However, the considerable overlap between Lyme disease and its mimics limits the clinical applicability of these predictive models. In practice, clinicians often assign an implicit risk for a given outcome in an unstructured manner as a part of their clinical decision-making,^12,13 referred to here as “clinician suspicion.” The ability of clinician suspicion to accurately identify children at either high or low risk for Lyme disease has not been rigorously evaluated.

To this end, we assembled a prospective cohort of patients undergoing evaluation for Lyme disease in 1 of the 5 participating emergency departments (EDs) located in Lyme disease–endemic areas. We queried the treating clinician’s suspicion of Lyme disease on a 10-point scale and determined the accuracy of this score for the final diagnosis of Lyme disease.

Methods

Study Design

We conducted a prospective cohort study at 5 participating EDs in Pedi Lyme Net (Boston Children’s Hospital [Boston, MA], Hasbro Children’s Hospital [Providence, RI], Nemours/Alfred I. duPont Hospital for Children [Wilmington, DE], Children’s Hospital of Philadelphia [Philadelphia, PA], and Children’s Hospital of Wisconsin [Milwaukee, WI]) between June 4, 2015, and July 31, 2017. The study enrollment period began on various dates for each participating site.

The study protocol was approved by the institutional review board of each participating institution with permission for data sharing. We obtained written informed consent from participating caregivers with patient assent as per institutional requirements. For the parent study, consent included permission to collect a research blood sample, review laboratory results, and perform telephone follow-up to determine clinical outcome.

Study Patients

We prospectively identified children aged ≥1 and ≤21 years who were undergoing ED evaluation for Lyme disease because they had an EM rash or Lyme disease serology ordered by the treating clinician. The diagnosis of EM rash was made by the treating clinicians by using the Centers for Disease Control and Prevention surveillance definition for an EM lesion. Clinicians were taught that a single primary EM lesion should be ≥5 cm in diameter with or without central clearing and that a lesion <5 cm in maximal diameter qualified for the diagnosis of EM only if each of the following were present: (1) it develops at the site of the known or suspected tick bite, (2) a time interval between the bite and the onset of the lesion is reported, and (3) the lesion is enlarging.¹⁴ Previously enrolled patients were eligible for enrollment during subsequent ED encounters if the treating clinician again obtained Lyme serology for clinical reasons.

Data Collection

At each of the participating sites, all clinicians had standardized training on study procedures either in person or electronically. After patient enrollment, the treating clinician completed standardized case forms to collect patient history, including duration of symptoms as well as physical examination findings. We also collected data from caregivers on whether the patient had been previously diagnosed with Lyme disease and dichotomized the response as follows: yes (possible or probable history of Lyme disease) or no (no history of previous Lyme disease). One month after enrollment, study staff at each participating site abstracted results of all Lyme disease 2-tiered serology performed within 30 days of enrollment as well as details of treatment provided.

Clinical Suspicion

At the time of patient enrollment, the treating attending provider was asked to estimate the probability that the patient had Lyme disease on a 1 to 10 scale, with 1 being “not likely to be Lyme disease” and 10 being “very likely to be Lyme disease.” Lyme disease serology results were not available at the time of the clinician suspicion assessment. For every attending clinician who estimated clinician suspicion for Lyme disease, we abstracted the unique attending provider identity.

Outcome Measure

Our primary outcome was the diagnosis of Lyme disease. We defined a case of Lyme disease as a patient with potential exposure to the causative organism who had either a physician-diagnosed EM lesion or positive 2-tiered serology results as well as manifestations compatible with Lyme disease.¹⁵ Because each of the 5 participating centers were located in a Lyme disease–endemic area, we assumed that every enrolled child had a potential exposure. We considered the following clinical manifestations compatible with Lyme disease by stage: early (EM lesion), early disseminated (multiple EM lesions, cranial neuritis, headache and/or neck pain or stiffness, electrocardiogram changes suggestive of carditis), or late (arthritis).¹ Enrolled patients with nonspecific symptoms (eg, fever or fatigue without any of the specific manifestations of early, early disseminated, or late Lyme disease) were classified as not having Lyme disease regardless of Lyme serology results. We included any 2-tiered serology performed within 30 days of enrollment to capture children who later seroconverted.

We defined a positive 2-tiered serology result as a positive or equivocal first-tier enzyme-linked immunosorbent assay (ELISA) test result followed by a positive immunoblot test result. Participating centers routinely used a variety of clinical Lyme disease serology assays: first-tier ELISA (4 centers used whole-cell sonicate [WCS] ELISA; 2 centers used MarDx [Trinity Biotech, Jamestown, NY], 1 center used VIDAS, 1 center used Zeus Scientific, and 1 center used VLsE ELISA [Zeus Scientific, Branchburg, NJ]) and second-tier immunoblots (3 centers used MarDx and 2 centers used Viramed). All immunoblots were interpreted by the clinical laboratory according to the Centers for Disease Control and Prevention recommendations.¹⁵ A positive immunoglobulin M immunoblot test result alone was considered positive only if the duration of symptoms was ≤30 days.^16,17 A positive immunoglobulin G immunoblot test result alone in a patient with symptoms compatible with Lyme disease was considered positive regardless of the previous Lyme disease history.

Statistical Analysis

We first compared children with Lyme disease to those without using the Mann–Whitney U test to compare continuous variables and the χ² test for proportions. For patients with an EM lesion, Lyme disease can be diagnosed by the treating clinician without Lyme disease serology results. Therefore, we measured the association between clinician suspicion and the diagnosis of Lyme disease using binary logistic regression after exclusion of children with an EM lesion. We then used a generalized estimating equation to measure this association after adjusting for likely confounders, such as attending provider identity, patient age, month and year of presentation, and clinical stage (early disseminated and late) as well as clustering by hospital center. Third, we constructed receiver operating characteristic (ROC) curves to depict the ability of clinicians to distinguish between children with and without Lyme disease. We plotted true-positives (sensitivity) versus false-positives (1 − specificity) for clinician suspicion scores from 1 to 10. We used the area under the curve (AUC) to quantify the discriminative ability of clinicians. On the basis of published standards, we defined AUCs of <0.7 as having poor discriminatory value, AUCs of 0.7 to 0.8 as minimally accurate, AUCs of 0.8 to 0.9 as having good accuracy, and AUCs of >0.9 as having excellent accuracy.¹⁸ Last, we examined the prevalence of Lyme disease on the basis of the clinician suspicion category. For this analysis, we categorized the clinician suspicion score as follows: unlikely (score 1–3), possible (4–7), or very likely (8–10).

We used SPSS software version 23.0 for all statistical analyses (IBM SPSS Statistics, IBM Corporation, Armonk, NY).

Results

Of the 3152 children undergoing evaluation for Lyme disease during the study period at the participating institutions, 1414 (44.9% of those eligible) were approached. Of those approached, 1021 participating caregivers and/or patients (72.2%) consented to study participation (Fig 1). The median patient age was 9 years (interquartile range, 5–13 years), and 564 (55.2%) were male patients. Enrolled children presented with clinical manifestations compatible with the following stages of Lyme disease: EM lesion only (n = 42; 4.1%), early disseminated disease (n = 479; 46.9%), and late disease (n = 467; 45.8%). The remaining children had nonspecific symptoms (n = 33; 3.2%).

FIGURE 1.

Study patients.

Overall, 238 children (23.3% of those enrolled) had Lyme disease. The proportion of children with Lyme disease varied between participating centers (range, 16.9%–36.4%). The Lyme disease diagnosis was made as follows: EM lesion alone (n = 27, 11.3% of Lyme disease patients), EM lesion plus positive 2-tiered serology result (n = 15, 6.3%), and positive 2-tiered serology result alone (n = 196, 82.4%). Of the 26 children who had repeat Lyme disease serology within 30 days of enrollment (2.5% of the study population), 3 seroconverted after the initial ED encounter. None of the children with nonspecific symptoms had a positive 2-tiered Lyme disease serology result. Of the 933 children with previous history of Lyme disease documented, 166 children (17.8%) had possible or probable Lyme disease previously, of which 53 met the study Lyme disease case definition.

We then compared children with Lyme disease to those without Lyme disease (Table 1). Children with Lyme disease were older and more likely to be male. More children were enrolled during the peak Lyme disease season (June through October), although Lyme disease was diagnosed throughout the year (Fig 2).

TABLE 1.

Comparison Between Study Patients With and Without Lyme Disease

Patient Characteristics	Lyme Disease (N = 238)	Not Lyme Disease (N = 783)	P
Age, ya	9 (6–13)	9 (5–13)	.74
Male sex, n (%)	143 (60.1)	418 (53.4)	<.001
Race, n (%)b			.001
White	208 (87.4)	518 (66.2)
African American	12 (5.0)	84 (10.7)
Other	13 (5.5)	80 (10.2)
Hispanic ethnicity, n (%)c	22 (9.2)	117 (14.9)	.017
Clinical presentation, n (%)			<.001
EM lesion	42 (17.6)	n/a
Early disseminated	82 (34.5)	397 (50.7)
Late	114 (47.9)	353 (45.1)
Nonspecific	n/a	33 (4.2)
Pretreated with antibiotics, n (%)	103 (43.3)	256 (32.7)	.013
Previous Lyme disease, n (%)	53 (22.3)	113 (14.4)	.001
Recent known tick bite, n (%)d	37 (15.5)	64 (8.2)	.003
Hospitalized, n (%)	75 (31.5)	185 (23.6)	.012

n/a, not applicable.

^aMedian (interquartile range).

^bMissing race (n = 106).

^cMissing ethnicity (n = 89).

^dMissing previous tick bite (n = 115).

FIGURE 2.

Frequency of Lyme disease diagnosis by month of presentation.

Clinician suspicion was documented by the enrolling attending physician for 1013 children (99.2% of those enrolled). Of the 216 attending physicians who recorded suspicion scores, 160 (74%) enrolled >1 child (range, 1–28 enrollments per provider). After exclusion of the 42 children with EM lesions, increasing clinician suspicion was associated with Lyme disease diagnosis (odds ratio, 1.44; 95% confidence interval [CI], 1.35–1.55). After adjustment for provider identity, patient age, month and year of presentation, and clinical stage as well as clustering by hospital center, a higher clinician suspicion score was associated with increased odds of Lyme disease (adjusted odds ratio, 1.45; 95% CI, 1.35–1.55). That is, the odds of Lyme disease increased by 45% for each 1-point increase in the clinician suspicion score.

For the 974 children without a single EM lesion, we then used ROC curve analysis to measure the overall ability of clinician suspicion to diagnose Lyme disease. Overall, clinician suspicion had minimal accuracy for the diagnosis of Lyme disease (Fig 3; AUC, 0.75; 95% CI, 0.71–0.79). Last, we examined the prevalence of Lyme disease by clinician suspicion category. Of the 554 children unlikely to have Lyme disease, 65 (11.7%) had Lyme disease, and of the 127 children likely to have Lyme disease, 39 (30.7%) did not have Lyme disease (Fig 4).

FIGURE 3.

ROC for clinician suspicion (score from 1 to 10) for the diagnosis of Lyme disease.

FIGURE 4.

Lyme disease diagnosis by clinician suspicion score for the 1013 children with clinician suspicion recorded by the treating attending provider.

Discussion

We enrolled a prospective cohort of >1000 children undergoing evaluation for Lyme disease at 1 of the 5 participating EDs. The treating clinician’s suspicion for Lyme disease assessed before Lyme serology results were available had only minimal accuracy for the diagnosis of Lyme disease. The strength of association was unchanged after adjustment for demographic and clinical factors associated with Lyme disease, suggesting that clinicians have accurately accounted for these factors in their risk assessments. Because a substantial minority of children in the low-risk group had Lyme disease and a substantial minority in the high-risk group did not have Lyme disease, confirmatory Lyme disease testing should always be used to guide final patient management.

Clinical impression can play a role in the initial management decisions for children presenting to the ED. In previous investigations, researchers have evaluated the accuracy of clinician suspicion for the emergency evaluation of children with blunt head trauma,¹⁹ abdominal trauma,²⁰ and fever.²¹ For children with head or torso trauma, clinician suspicion had lower sensitivity but higher specificity for clinically important injury than corresponding clinical prediction rules.¹⁹ For febrile infants, neither clinician suspicion nor a structure observation scale (eg, Yale Observation Scale score) effectively identified infants at low risk of bacterial infection, and these should not be used to guide clinical decision-making.²¹ With our study, we are the first to examine prospectively collected clinician suspicion of Lyme disease in a cohort of children undergoing ED evaluation for Lyme disease. Because accuracy did not vary by presentation, treating physicians appear to have accurately incorporated clinical factors in their initial risk assessment. However, given the minimal accuracy overall, additional decision-support tools combined with accurate and rapid diagnostic testing would improve initial care for children with potential Lyme disease.

Physicians evaluating children with potential Lyme disease must make initial management decisions before results of Lyme disease tests are available. For example, children with Lyme disease facial palsy are treated with appropriate antibiotics, whereas those with idiopathic facial palsy (eg, Bell’s palsy) may recover more rapidly with corticosteroid therapy.^22,23 Although evidence is limited, corticosteroids might slow recovery for those with Lyme disease facial palsy.²⁴ Alternately, a child with inflammatory arthritis could have Lyme disease or septic arthritis (ie, bacterial infection of the joint fluid).^9,10 Whereas children with Lyme arthritis are initially treated with oral antibiotics, those with septic arthritis require joint aspiration and potential irrigation as well as parenteral antibiotics. Although clinician suspicion of Lyme disease can assist initial management, with our findings we suggest that final decisions should await confirmatory testing to avoid both over- and underdiagnosis of Lyme disease.

For children with symptoms compatible with early disseminated or late disease, the diagnosis of Lyme disease is based on 2-tiered serology. As demonstrated by the testing protocols currently in place in the 5 participating study institutions, a variety of first-tier Lyme disease tests are available and in clinical use. The C6 Lyme ELISA measures antibody reactivity to VlsE, a highly conserved surface protein of the causative Borrelia organism.²⁵ In adults with early Lyme disease, the C6 ELISA had higher sensitivity but similar specificity when compared to 2-tiered testing with the WCS ELISA as a first-tier test.^26,27 In children, C6 ELISA alone had similar sensitivity with but higher specificity when compared to WCS ELISA alone.²⁸ Because the currently available C6 ELISA could be run by a diagnostic laboratory in as little as 1 hour,²⁹ the C6 ELISA combined with clinician suspicion could help guide initial clinical decision-making. Well-recognized limitations with currently available Lyme disease serology tests that include false-negatives in early disease and false-positives after previous infection^18,30,31 should not be forgotten. Among our study patients with Lyme disease, 3 participants seroconverted within 30 days of enrollment and 53 participants had a previous history of possible or probable Lyme disease. New approaches, which include metabolomics³² and measuring host response to infection,³³ may ultimately improve diagnostic accuracy but are not yet available clinically.

Our study had several limitations. First, each of our participating clinical sites were located in a Lyme disease–endemic area, and our findings should not be applied to nonendemic regions. Second, we enrolled only approximately one-third of the potentially eligible children. The majority of missed eligible patients presented to the ED when study staff were not available, although a minority of caregivers refused study participation. Third, we did not capture patient-level Lyme disease exposure (eg, location of residence or amount of outdoor exposure), although clinicians likely incorporated these epidemiologic factors in their risk assessments. Fourth, despite practicing in endemic areas, enrolling providers had varying knowledge and experience with Lyme disease. We did require that the most experienced clinician, the treating attending provider, provide the clinician suspicion score. Study participation may have had the unintended consequence of providing additional Lyme disease clinical education to the enrolling providers. However, clinician suspicion performance did not change after adjusting for either study year or after clustering by center. Fifth, although we included a large number of unique attending providers, the association between clinician suspicion and odds of Lyme disease diagnosis did not change after adjusting for provider identity. Sixth, we limited enrollment to children who were being evaluated clinically for Lyme disease, so we missed children with Lyme disease when the diagnosis was not considered by the treating provider. Last, we compared our clinician score against an imperfect Lyme disease gold standard. Although an EM lesion is sufficient for Lyme disease diagnosis, other common skin conditions such as cellulitis may provide a clinical mimic. Additionally, EM lesions can present atypically, leading to diagnostic confusion,^34–36 and we relied on the treating clinician’s diagnosis. Two-tiered Lyme disease serology results can be negative in early Lyme disease³⁷ and can remain positive after previous infection.³⁸ To limit the misclassification of children with early disseminated Lyme disease, we reviewed all Lyme disease serology performed within a month of study enrollment. Additionally, one-fifth of children had Lyme disease previously, potentially complicating the interpretation of Lyme disease serology. Despite these well-described flaws, our Lyme disease case definition mimics current clinical practice and represents the most accurate currently available clinical case definition available. New approaches for the diagnosis of Lyme disease that include more rapid and more accurate diagnostic tests are needed.

Conclusions

Children with a classic EM lesion can be diagnosed clinically with Lyme disease without further testing. For other children with potential Lyme disease, clinician suspicion was only minimally accurate in our multicenter prospective cohort. Although 2-tiered Lyme disease serology takes several days to return results in most clinical settings, final patient management decisions should await confirmatory test results to avoid both over- and underdiagnosis of Lyme disease.

Glossary

AUC

area under the curve

CI

confidence interval

ED

emergency department

ELISA

enzyme-linked immunosorbent assay

EM

erythema migrans

ROC

receiver operating characteristic

WCS

whole-cell sonicate