Accuracy of current diagnostic criteria for acute bacterial infection in older emergency department patients

Abstract

Objectives

To evaluate the accuracy of emergency department (ED) physicians’, the Loeb criteria, and CDC guideline diagnoses of acute bacterial infection in older adults compared to a gold standard expert review.

Design

Prospective, observational study.

Setting

Urban, tertiary-care ED.

Participants

ED patients aged ≥65 years, excluding those incarcerated, traumas, non-English speaking, or unable to consent.

Measurements

Two physician experts identified bacterial infections using clinical judgement, patient surveys, and medical records; a third adjudicated in cases of disagreement. Agreement and test characteristics were measured for ED physician diagnosis, Loeb criteria, and CDC surveillance guidelines.

Results

Gold-standard review identified bacterial infection in 77/424 patients (18%): 18(4.2%) lower respiratory, 19(4.5%) urinary tract (UTI), 22(5.2%) gastrointestinal, and 15(3.5%) skin/soft tissue. ED physicians diagnosed infection in 71 (17%), but there were 33 with under- and 27 with over-diagnosis. Physician agreement with the gold standard was moderate for infection overall and each infection type (kappa [k] 0.48–0.59), but sensitivity was low (<67%), and negative likelihood ratio (LR[−]) >0.30 for all infections. Loeb criteria had poor sensitivity, agreement, and LR(−) for lower respiratory (50%; k=0.55; 0.51) and UTI (26%; k=0.34; 0.74), but 87% sensitivity (k=0.78; LR[−] 0.14) for skin/soft tissue infection. CDC guidelines had moderate agreement, but poor sensitivity and LR(−).

Conclusions

Infections are often under- and over-diagnosed by emergency physicians in older adults. The Loeb criteria are useful only for diagnosing skin/soft tissue infections. CDC guidelines are inadequate in the ED. New criteria are needed to aid ED physicians in accurately diagnosing infection in older adults.

INTRODUCTION

Older adults visit U.S. emergency departments (EDs) 3 million times yearly for an infectious disease-related problem, resulting in 1.7 million annual infection-related admissions and >120,000 annual deaths.¹ ED care of older adults with infection poses a substantial diagnostic challenge given the frequent absence of typical signs and symptoms, the unavailability of culture results, and the frequent presence of chronic bacterial colonization.^2–9 This can result in either a failure to recognize the presence of acute bacterial infection (under-diagnosis) or inappropriately attributing acute illness to a bacterial infection (over-diagnosis).⁶ Under-diagnosis leads to failure to appropriately treat infections.^10,11 Over-diagnosis can result in failure to identify other acute medical conditions and unnecessary overuse of antibiotics.^12,13 Over-diagnosis rates have been as high as 43% for urinary tract infection (UTI) and 27% for pneumonia in older ED patients.^6,14

ED physician diagnosis of infection in older adults often disagrees with inpatient physicians’ diagnoses.¹⁵ One group found ED over-diagnosis of community acquired pneumonia compared to inpatient diagnosis, although accuracy of the inpatient diagnosis was not assessed.⁶ There are no validated diagnostic criteria for ED use in the older population. Most criteria, such as those from the Centers for Disease Control (CDC) or Infectious Disease Society of America, rely on culture results which are not available in the ED.^16,17 Loeb et al used expert opinion to develop criteria for initiating antibiotics in long term care facilities (LTCFs)(Supplemental Table 1).⁷ Although the Loeb criteria mirror ED care as they are used prior to return of culture results, they have not been studied in the ED. A primary goal of both the CDC guidelines and Loeb criteria is to maximize specificity of diagnosis. This is either to establish conservative estimates for surveillance purposes (CDC) or to decrease LTCF antibiotic use (Loeb). Conversely, ED physicians are most concerned with sensitivity to avoid missing infections. As a result, the usefulness and accuracy of these two criteria in the ED is not known.

The goal of this study was to determine the accuracy of current bacterial infection diagnoses when compared to gold standard experts in older ED patients using sensitivity, specificity, and likelihood ratios. The objectives were to identify the accuracy of the ED physician, Loeb Criteria, and CDC guidelines in diagnosing acute bacterial infection. We also sought to identify agreement between the gold standard experts and the ED physician, Loeb criteria, and CDC guidelines. We hypothesized that ED physician diagnoses of acute bacterial infection demonstrates no better than fair agreement with the gold standard for bacterial infection. and that the Loeb criteria and CDC guidelines demonstrate good or very good agreement with the gold standard.

METHODS

Study design and setting

We conducted a prospective, observational study of older adults presenting to an urban, tertiary care ED with 60,000 annual visits and 5% older adults. The department is staffed by board-certified emergency medicine attendings and emergency medicine residents. Patients were enrolled from 10/2011 through 04/2013. We obtained institutional review board approval and followed STROBE guidelines.¹⁸

Participants

All ED patients aged ≥65 years were potentially eligible. Exclusion criteria included incarceration, primary evaluation by trauma team, suicidal or homicidal ideation, prior enrollment, non-English speaking, or patient otherwise unable to complete the survey.

Conduct of the study

Study personnel reviewed in real-time the ED electronic medical record (IBEX PulseCheck [PICIS Clinical Solutions, Inc, Wakefield MA] through 09/2012; EPIC [EPIC Systems, Verona, WI] afterwards) of consecutive patients ≥65. Personnel were available 4–6 hours per day, generally during normal business hours, but with 20% evening shifts and 10% weekends. Potentially eligible patients underwent a brief mental status screen (the Confusion Assessment Method-ICU [CAM-ICU]) to determine capacity. Presence of dementia was determined by chart review and direct query of patient or accompanying persons. In patients with history of dementia and/or positive CAM-ICU, consent was obtained via a legally authorized representative.^19–21

Patient’s were surveyed in the ED regardingdemographics; fever within the prior 24 hours; rigors; confusion; malaise; and urinary, respiratory, skin, neurologic, orthopedic and gastrointestinal symptoms. Chart information included: demographics, chief complaint, medical history for Charlson Comorbidity index, ED disposition, ED vital signs, physical exam findings related to infection, indwelling devices, laboratory results, microbiology results, and imaging reports. We considered an element absent if it was not noted in the ED chart.²² For these data, only ED nursing, advanced practice provider, and physician documentation was used. Abstractors were trained research staff who were not blinded to study hypotheses. A code book was used and abstractors underwent an initial two hour training session.

Upon final ED disposition, study staff administered a survey to the attending ED physician or senior resident querying the physician’s impression of the likelihood of an acute bacterial infection and the infections suspected on a 5 point Likert scale from Very Unlikely to Very Likely. ED physician impression was based only upon information available at the time of ED disposition. For admitted patients, a similar survey was provided to the inpatient attending physician on hospital day 5 or at discharge for patients discharged earlier. Suspicion of infection was based only on physician impression. Physicians were not provided with diagnostic criteria on diagnosing infection in older adults.

Because of the lack of a gold standard test for some infections (e.g. UTI) and the possible lack of symptoms in older adults for others, we used agreement among expert case reviewers as the gold standard for presence of infection.^23–26 Two expert physicians, one board-certified in infectious disease (RL) and one board-certified in emergency medicine and internal medicine with expertise in geriatrics (JMC), reviewed the study surveys, the ED visit, any subsequent inpatient stay, and any healthcare visit for 10 days after the ED visit. They had access to patient report of symptoms through the patient survey, all provider notes, and all laboratory, microbiology, and radiology reports. Experts identified the presence of individual bacterial infections based on a 5 point Likert scale. In cases where experts disagreed, a third expert board-certified in emergency medicine and graduate of a geriatrics fellowship (LTS) performed a review. Adjudicators were blinded to others’ determinations. The experts were familiar with IDSA definitions, but used clinical judgment rather than specific definitions to identify infections. JMC reviewed the Loeb and CDC guidelines to determine if subjects met their criteria.^7,16

Measured variables

Bacterial infection

Present if one of the following was identified: lower respiratory tract (pneumonia or empyema, but excluding isolated bronchitis or acute COPD exacerbations), UTI (cystitis, pyelonephritis, or prostatitis), skin/soft tissue (cellulitis, cutaneous abscess, surgical wound infection, deep skin infection, diabetic foot infection, or decubitus ulcer), neurologic (bacterial meningitis), orthopedic (osteomyelitis or septic arthritis), gastrointestinal (appendicitis, diverticulitis, hepatobiliary infection, colitis, intraabdominal abscess, or other intra-abdominal), or isolated bacteremia.

Gold standard presence of bacterial infection

Likert scale score of 4 or 5 for any bacterial infection by at least two expert reviewers.

Presence of any acute bacterial infection, ED physician and subsequent treating physician

Suspicion of at least one bacterial infection as either a 4 or 5 (likely or very likely) by the relevant attending physician. The presence of specific infection types was also noted.

Presence of lower respiratory tract, urinary tract, or skin/soft tissue infection, Loeb LTCF criteria:⁷

The Loeb criteria include definitions only for lower respiratory tract infection, UTI and skin/soft-tissue infection (Supplemental Table 1).

Presence of any acute bacterial infection, CDC surveillance guideline definitions:¹⁶

Infections are defined in the CDC document. They generally include microbiologic confirmation in the setting of signs and symptoms.

Data analysis

We calculated descriptive statistics including mean with standard deviation and proportions with 95% confidence intervals (CI). For comparisons of diagnosis we identified agreement using Cohen’s kappa with 95% CI. We used commonly accepted cutoffs to define fair (k=0.41–0.60), good (k=0.61 – 0.80) and very good (k >0.80) agreement.²⁷ Because kappas in a 2×2 table may have maximum values <1.0, we also calculated the maximum possible kappa.²⁸ We calculated sensitivity, specificity, positive likelihood ratio (LR[+]) and negative likelihood ratio (LR[−]) with 95% CI for each comparison. All analyses were performed with SAS v9.4(SAS Institute, Inc., Cary, NC).

The primary outcome in each comparison was the presence of any acute bacterial infection. Secondary outcomes included the presence of specific infection types. We made three comparisons to the gold standard: 1) the ED physician diagnosis, 2) the Loeb LTCF criteria, and 3) the CDC guidelines.

Assuming a kappa of 0.7 (good agreement), a 15% true acute infection rate, and alpha of 0.05, we planned for 397 subjects to calculate kappa with a confidence interval ranging from 0.6 to 0.8 and thus allow confirmation of the presence of good agreement.

RESULTS

We enrolled 439 patients. Fifteen were excluded, leaving 424 subjects for analysis (Figure 1). Study population characteristics are displayed in Table 1. According to our gold standard, 77 (18%) had a bacterial infection including 18 (4.2%) lower respiratory, 19 (4.5%) UTI, 22 (5.2%) gastrointestinal, and 15 (3.5%) skin/soft tissue. Agreement of gold standard reviewers for infection was: 95% for any infection, 99% lower respiratory, 97%, UTI, and 98% skin/soft tissue. ED physicians diagnosed bacterial infection in 71 (17%). The Loeb LTCF criteria identified 14 (3.3%) with lower respiratory infection, 13 (2.1%) with UTI, and 15 (3.5%) with skin/soft tissue infection. The CDC guidelines identified infection in 43 (10%). Even where proportions of infections were similar, there were substantial differences between groups over which patients had infection.

Figure 1.

Flow diagram of study inclusion of older adults presenting to the ED

*For approximately three months at the beginning of the study period, we attempted to enroll only nursing home patients. This criterion was then modified. No study patients were enrolled during that time period.

Table 1.

Characteristics of 424 older adult emergency department patients

Characteristic	n	Proportion/mean	95% CI/SD
Demographics: Mean age	—	74 years	7.4
Female Gender	244	58%	(53–62)
Race: Black	71	17%	(13–21)
White	344	81%	(77–85)
Other	9	2.0%	(1.0–4.0)
Hispanic ethnicity	7	1.6%	(0.7–3.4)
Delirium	16	3.8%	(2.2–6.0)
Dementia	5	1.2%	(0.4–2.7)
Patient-reported symptoms
Patient-reported symptoms: Fever <24 hours	30	7.1%	(4.8–9.9)
Rigors	68	16%	(13–20)
Confusion/altered mental status	42	9.9%	(7.2–13)
Malaise/lethargy/fatigue	183	43%	(38–48)
ED disposition: Discharge	173	41%	(36–46)
Admit to intensive care	4	1.0%	(0.2–2.3)
Admit inpatient unit	245	58%	(53–62)
Left against medical advice/expired	1	0.2%	(0.0–1.3)
ED process: Antibiotics ordered in the ED	78	18%	(15–22)
Vital signs: Initial temperature >38.0° C	6	1.4%	(0.5–3.0)
Highest temperature >38.0° C	10	2.4%	(1.1–4.3)
Initial heart rate	—	84 bpm	20
Initial systolic blood pressure	—	138 mmHg	28
Initial respiratory rate	—	18/minute	4
Confusion/altered mental status	25	5.9%	(3.8–8.6)
Malaise, lethargy, or fatigue	79	19%	(15–23)
Charlson comorbidity index score		2.3
Presence of infection-gold standard
Any bacterial infection	77	18%	(15–22)
Lower respiratory	18	4.2%	(2.5–6.6)
Urinary tract	19	4.5%	(2.7–6.9)
Skin and soft tissue	15	3.5%	(2.0–5.8)
Gastrointestinal	22	5.2%	(3.3–7.8)
Orthopedic	1	0.2%	(0.0–1.3)
Neurologic	0	0.0%	(0.0–0.9)
Presence of infection - ED physician
Any bacterial infection	71	17%	(13–21)
Lower respiratory	28	6.6%	(4.4–9.4)
Urinary tract	24	5.7%	(3.7–8.3)
Skin soft tissue	18	4.3%	(2.5–6.6)
Gastrointestinal	18	4.2%	(2.5–6.6)
Orthopedic	2	0.5%	(0.0–1.7)
Neurologic	1	0.2%	(0.0–1.3)
Presence of infection - Loeb guidelines
Lower respiratory	14	3.3%	(1.8–5.5)
Urinary tract	13	2.1%	(1.6–5.2)
Skin soft tissue	15	3.5%	(2.0–5.8)
Presence of infection - CDC guidelines
Any bacterial infection	43	10.0%	(7.4–13)
Lower respiratory	11	2.6%	(1.3–4.6)
Urinary tract	5	1.2%	(0.4–2.7)
Skin soft tissue	10	2.4%	(1.1–4.3)
Gastrointestinal	15	3.5%	(2.0–5.8)
Orthopedic	0	0.0%	(0.0–0.9)
Neurologic	0	0.0%	(0.0–0.9)

CI=confidence interval, SD=standard deviation, ED=emergency department.

Table 2 shows agreement between the ED physician and gold standard. Agreement was moderate for presence of infection overall and for each infection type (kappa 0.49–0.59). Both over-diagnosis and under-diagnosis were common. ED physicians and the gold standard agreed on presence of infection in 44 patients. In 33 cases the ED physician missed the infection and in 27 cases thought an infection was present which was not confirmed by the review. Results for individual infection types were similar. Table 3 demonstrates test characteristics of ED physician diagnosis overall and by specific types. ED physician sensitivity and LR(−) were poor for infection overall and by infection type, but specificity and LR(+) were high.

Table 2.

Agreement between ED physician diagnosis and gold standard expert review for presence of acute bacterial infection in older emergency department patients

a) Any bacterial infection
	Gold standard diagnosis
ED physician diagnosis	Present	Absent	Kappa (95%CI)	Maximum kappa
Present	44	27	0.51 (0.40, 0.62)	0.95
Absent	33	320
b) Lower respiratory infection
	Gold standard diagnosis
ED physician diagnosis	Present	Absent	Kappa (95%CI)	Maximum kappa
Present	12	16	0.50 (0.31, 0.68)	0.77
Absent	6	390
c) Urinary tract infection
	Gold standard diagnosis
ED physician diagnosis	Present	Absent	Kappa (95%CI)	Maximum kappa
Present	11	13	0.49 (0.30, 0.67)	0.88
Absent	8	392
d) Skin and soft tissue infection
	Gold standard diagnosis
ED physician diagnosis	Present	Absent	Kappa (95%CI)	Maximum kappa
Present	10	8	0.59 (0.39, 0.79)	0.91
Absent	5	401
e) Gastrointestinal infection
	Gold standard diagnosis
ED physician diagnosis	Present	Absent	Kappa (95%CI)	Maximum kappa
Present	11	7	0.53 (0.34, 0.72)	0.90
Absent	11	395

Table 3.

Test characteristics of each method of diagnosis compared to the gold standard diagnosis of infection in older emergency department patients

	Sensitivity (95% CI)		Specificity (95% CI)		LR(+) (95% CI)		LR(−) (95% CI)
ED physician
Any bacterial infection	0.57	(0.45, 0.68)	0.92	(0.89, 0.95)	7.34	(4.32, 10.37)	0.46	(0.34, 0.58)
Lower respiratory	0.67	(0.41, 0.87)	0.96	(0.94, 0.98)	16.92	(7.06, 26.77)	0.35	(0.12, 0.57)
Urinary tract	0.58	(0.34, 0.80)	0.97	(0.94, 0.98)	18.04	(6.13, 29.94)	0.44	(0.20, 0.66)
Skin and soft tissue	0.67	(0.38, 0.88)	0.98	(0.96, 0.99)	34.08	(7.63, 60.54)	0.34	(0.096, 0.58)
Gastrointestinal	0.50	(0.28, 0.72)	0.98	(0.96, 0.99)	28.71	(4.38, 53.05)	0.51	(0.29, 0.72)
Orthopedic	0.00	(0, 0.98)	1.00	(0.98, 1.00)	–	–	–	–
Loeb criteria
Lower respiratory	0.50	(0.26, 0.74)	0.99	(0.97, 1.00)	40.60	(0.44, 80.75)	0.51	(0.27, 0.74)
Urinary tract	0.26	(0.09, 0.51)	0.99	(0.97, 1.00)	26.64	(−6.27, 59.56)	0.74	(0.54, 0.94)
Skin and soft tissue	0.87	(0.60, 0.98)	0.99	(0.97, 1.00)	70.89	(7.36, 134.43)	0.14	(−0.04, 0.31)
CDC guideline
Any bacterial infection	0.53	(0.42, 0.65)	0.99	(0.98, 1.00)	92.38	(−37.17, 221.94)	0.47	(0.36, 0.58)
Lower respiratory	0.56	(0.31, 0.78)	1.00	(0.99, 1.00)	225.56	(−227.53, 678.64)	0.45	(0.21, 0.68)
Urinary tract	0.26	(0.09, 0.51)	1.00	(0.99. 1.00)	–	–	–	–
Skin soft tissue	0.60	(0.32, 0.84)	1.00	(0.99, 1.00)	245.40	(−247.56, 738.36)	0.40	(0.15, 0.65)
Gastrointestinal	0.68	(0.45, 0.86)	1.00	(0.99, 1.00)	–	–	–	–
Orthopedic	0.00	(0, 0.98)	1.00	(0.99,1.00)	–	–	–	–

LR(+)=positive likelihood ratio, LR(−)=negative likelihood ratio, CI=confidence interval, ED=emergency department, CDC=Centers for Disease Control

Table 4 demonstrates agreement between the Loeb criteria and the gold standard expert review. Agreement was best for skin/soft tissue infection (kappa 0.78). Table 3 displays test characteristics which are notable for poor sensitivity and inadequate LR(−) for lower respiratory and UTI, but sensitivity of 87% (95% CI, 60–98%) and LR(−) 0.14 for skin/soft tissue infection. Specificity and LR(+) were high for all three infections.

Table 4.

Agreement of the Loeb criteria and CDC infection surveillance guidelines with gold standard expert review for presence of acute bacterial infection in older emergency department patients

		Gold standard diagnosis
a) Loeb - Lower respiratory infection		Present	Absent	Kappa(95%CI)	Maximum Kappa
Loeb Criteria Diagnosis:	Present	9	5	0.55	0.87
	Absent	9	401	(0.33, 0.76)
		Gold standard diagnosis
b) Loeb – Urinary tract infection		Present	Absent	Kappa(95%CI)	Maximum Kappa
Loeb Criteria Diagnosis:	Present	5	4	0.34	0.63
	Absent	14	401	(0.11, 0.57)
		Gold standard diagnosis
c) Loeb – Skin and soft tissue infection		Present	Absent	Kappa(95%CI)	Maximum Kappa
Loeb Criteria Diagnosis:	Present	13	5	0.78	0.91
	Absent	2	404	(0.62, 0.94)
		Gold standard diagnosis
d) CDC - Any bacterial infection		Present	Absent	Kappa(95%CI)	Maximum Kappa
CDC guideline diagnosis:	Present	41	2	0.64	0.67
	Absent	36	345	(0.53, 0.74)
		Gold standard diagnosis
e) CDC - Lower respiratory infection		Present	Absent	Kappa(95%CI)	Maximum Kappa
CDC guideline diagnosis:	Present	10	1	0.68	0.75
	Absent	8	405	(0.48, 0.88)
		Gold standard diagnosis
f) CDC - Urinary tract infection		Present	Absent	Kappa(95%CI)	Maximum Kappa
CDC guideline diagnosis:	Present	5	0	0.40	0.40
	Absent	14	405	(0.16, 0.65)
		Gold standard diagnosis
g) CDC - Skin and soft tissue infection		Present	Absent	Kappa(95%CI)	Maximum Kappa
CDC guideline diagnosis:	Present	9	1	0.72	0.79
	Absent	6	408	(0.51, 0.92)
		Gold standard diagnosis
h) CDC – Gastrointestinal infection		Present	Absent	Kappa(95%CI)	Maximum Kappa
CDC guideline diagnosis:	Present	15	0	0.80	0.80
	Absent	7	402	(0.66, 0.94)

Agreement between the CDC criteria and the gold standard expert review was k>0.60 for any infection, lower respiratory, skin/soft tissue, and gastrointestinal infections but was 0.40 for UTI (Table 4). Sensitivity was 53% (95% CI 42–65%) for any bacterial infection and was below 68% for all individual infection types. Specificity was high. Test characteristics are noted in Table 3.

DISCUSSION

The problem of accurate diagnosis of acute infection in older adults can be attributed to several factors including absence of classic symptoms, lack of culture data in the ED, overlap of symptoms among conditions, and frequent presence of chronic bacterial colonization.^2,5,29 The problems of both over-diagnosis and under-diagnosis of infection are potentially critical.¹⁴ Given the possibility of missing alternative diagnoses, spreading antimicrobial resistance, increasing Clostridium difficile, and other side effects of antibiotics, it is important to treat only patients with true acute infections.^13,30 Conversely, under-diagnosis may lead to delays in treatment, worsening of infection, extended hospital stays, and increased cost.^10,11 Acute bacterial infections were present in 18% of this cohort of older ED adults. This demonstrates the significant burden of infections in older ED adults and is consistent with prior estimates.^1,31

These issues also complicate the study of infections in older adults. Diagnostic criteria that rely on presence of symptoms and/or culture results may not be accurate in a population where symptoms are often absent and colonizing microorganisms often present.^3,5 For example, the high prevalence of asymptomatic bacteriuria in older adults means that relying solely on urine cultures overestimates acute UTIs.^4,26,32 Conversely, 50% of older adults with bacteremic UTI, a clear case of true acute infection, lack urinary symptoms.²⁹ As a result, a gold standard cannot be reliably identified based solely on culture results or patient symptoms.

By using a multidisciplinary expert gold standard review, we have extended a successful approach in other settings to the ED. This approach has been used in inpatients for skin/soft tissue infection and UTI, among other conditions.^23–26 Experts had access to patient survey responses on infection symptoms, laboratory and microbiology data, provider notes, and patient course. We believe this allowed them to consider all relevant factors and rates of agreement among reviewers were high.

The study population included all older ED adults so as to avoid missing those patients not initially identified as infected by the ED physician. The necessity of this approach was validated by the low sensitivity of ED physicians in diagnosing infection. Because we included many patients who clearly did not have infection, we had a large proportion of true negatives, resulting in specificities and LR(+)s which were generally higher than would be seen in a more discrete population. As a result, specificity and LR(+) results should be interpreted with caution. Commonly, LR(+)s >10 indicate usefulness for ruling in the diagnosis.³³ However, this should be interpreted cautiously as the very high specificities substantially increased the LR(+)s. Lower specificity might be seen in more discrete cohorts (e.g., lower respiratory infection in patients with dyspnea) and would move the LR(+) values closer to 1. Therefore, we are unable to conclude that either the emergency physician diagnosis or the criteria studied would substantially increase the post-test probability of disease in a more specific ED population.

Our approach did allow for accurate estimates of sensitivity and LR(−). A LR(−) of <−0.2 corresponds to a 30% decrease in posttest probability of disease, and only the Loeb criteria for skin/soft tissue infection had a value <0.2.³³ In a more specific patient population, sensitivity might be lower, but this would result in the LR(−) values moving closer to one. This would not affect our conclusions which have already determined the Loeb criteria and CDC guidelines are unable to impact post-test probability of absence of disease.

There was only fair agreement between the emergency physician and the gold standard. Emergency physician had poor sensitivity overall (57%) and for all infection types. Table 2 demonstrates both under-diagnosis and over-diagnosis as shown by the number of false positives. Others have found similar high rates of over-diagnosis between ED and inpatient diagnoses of community acquired pneumonia with 27% of those diagnosed in the ED receiving alternative discharge diagnoses.⁶ Gordon et al found that 43% of older women diagnosed with UTI in the ED had negative urine cultures.¹⁴ In our group, 27/71 older patients (38%) diagnosed with UTI were not thought to have UTI by the gold standard (over-diagnosis). These prior studies have only included patients diagnosed with infection in the ED, and therefore only addressed over-diagnosis (specificity). By enrolling all older ED adults, we also identified high rates of under-diagnosis in this age group.

The Loeb criteria represent, to our knowledge, the most relevant criteria to the ED setting as they were designed to be used in LTCFs prior to return of culture results. However, they were designed in part to prevent antibiotic overuse.^7,34 Among the three relevant infection types, agreement was good only for skin/soft tissue infections. The Loeb criteria failed to identify half of lower respiratory infections and three-fourths of UTIs. However, they did accurately diagnose skin/soft tissue infection, although confidence intervals were wide.

The CDC guidelines were created for infection surveillance and are designed to prevent false positives.¹⁶ As a result, specificity and LR(+) were high, but sensitivity and LR(−) were low. Agreement was high because of the lack of false positives. If a patient has infection by the CDC guidelines it is likely present. However sensitivity and LR(−) was so poor that these are not useful in guiding ED clinical care. As this study was ongoing, in 2012 the CDC published an update to LTCF surveillance definitions, the revisited McGeer criteria.³⁵ Applying these criteria to the study population resulted in minimal classification changes when compared to CDC guidelines.

Limitations include the use of expert review for the gold standard which could result in misclassification bias. But, this approach has been successful in other infectious disease settings and reviewer agreement was high. The expert review approach also avoids the pitfalls of infections diagnosis in older adults as experienced clinicians can account for factors such as atypical symptoms in ways that current algorithms and definitions do not. We are further exploring these expert diagnoses using Bayesian modeling in ongoing analyses. An additional limitation is the choice to enroll all older adults including many clearly without infection. Although this affected our test characteristics, it allowed us to identify sensitivities of the criteria. We did not enroll patients overnight or in equal amounts on day and evening shifts. Those patients could be systematically different than study patients. Also, some patients were not enrolled due to inability to consent, possibly resulting in selection bias. However, admission rates were high, indicating that ill patients were enrolled. The use of a single site could also affect results.

In conclusion, infections are present in 18% of ED patients ≥65 but are often both under- and over-diagnosed by ED physicians. ED physician sensitivity was low and LR(−) was not sufficient to substantially alter post-test probability of infection. Although specificity was high, this was primarily due to the nature of the population studied and emergency physicians over-diagnosed a number of patients. The Loeb LTCF criteria are sufficiently sensitive with adequate LR(−) to be used in diagnosing skin/soft tissue infections, but not for lower respiratory infection or UTIs. CDC infection surveillance guidelines are highly specific, but sensitivity was low and LR(−) inadequate for the ED. New criteria should be sought to aid ED physicians in accurately diagnosing infection in this population. Potential approaches include developing ED/acute care-specific clinical decision rules and diagnostic algorithms. In addition, biomarkers for infection, whether in serum, urine, or lung should be explored both alone and as supplements to other clinical measurements.³⁶

Supplementary Material

Supp Table S1

Supplementary Table S1: Loeb Minimum Criteria for Initiation of Antibiotics in Long Term Care Facility Residents