Performance of Ultrasound in the Diagnosis of Appendicitis in Children in a Multicenter Cohort

Abstract

Objectives

The objectives were to assess the test characteristics of ultrasound (US) in diagnosing appendicitis in children and to evaluate site-related variations based on the frequency of its use. Additionally, the authors assessed the test characteristics of US when the appendix was clearly visualized.

Methods

This was a secondary analysis of a prospective, 10-center observational study. Children aged 3 to 18 years with acute abdominal pain concerning for appendicitis were enrolled. US was performed at the discretion of the treating physician.

Results

Of 2,625 patients enrolled, 965 (36.8%) underwent abdominal US. US had an overall sensitivity of 72.5% (95% confidence interval [CI] = 58.8% to 86.3%) and specificity 97.0% (95% CI = 96.2% to 97.9%) in diagnosing appendicitis. US sensitivity was 77.7% at the three sites (combined) that used it in 90% of cases, 51.6% at a site that used it in 50% of cases, and 35% at the four remaining sites (combined) that used it in 9% of cases. US retained a high specificity of 96% to 99% at all sites. Of the 469 (48.6%) cases across sites where the appendix was clearly visualized on US, its sensitivity was 97.9% (95% CI = 95.2% to 99.9%), with a specificity of 91.7% (95% CI = 86.7% to 96.7%).

Conclusions

Ultrasound sensitivity and the rate of visualization of the appendix on US varied across sites and appeared to improve with more frequent use. US had universally high sensitivity and specificity when the appendix was clearly identified. Other diagnostic modalities should be considered when the appendix is not definitively visualized by US.

Acute abdominal pain accounts for 5% to 10% of all pediatric emergency department (PED) visits. Approximately one-third of children with acute abdominal pain concerning for appendicitis will be diagnosed with it, making appendicitis the most common surgical emergency in children.^1–3 The diagnosis of pediatric appendicitis can be difficult, with a substantial proportion misdiagnosed based on clinical features and laboratory tests alone.⁴ Computed tomography (CT) has become heavily relied on in the evaluation of children with possible appendicitis.^1,5 However, because of increasing concern over the long-term malignancy risk related to CT-associated ionizing radiation, its routine use is being reappraised.^6–9 There is increasing interest in the use of ultrasound (US) as the primary imaging modality for establishing or ruling out the diagnosis of appendicitis in children and reserving CT as a secondary diagnostic modality.^10–13 Recently, investigators noted a trend toward increased reliance on US and decreased use of CT for children with appendicitis among large U.S. pediatric hospitals.¹⁴

The current literature on the test characteristics of US for diagnosis of appendicitis is composed of mainly single-center studies, where US has been found to have varied sensitivity (78% to 100%) and specificity (88% to 98%).^{4,5,13,15–22} US is well known to be operator-dependent, yet there is scant information about its performance at centers using it infrequently.²³ Additionally, two retrospective studies suggest that the test characteristics of US appear to improve when restricted to patients where the appendix was clearly identified, but further data are needed in this regard.^24,25

We conducted this secondary analysis from a large multicenter cohort to assess the test characteristics of US in diagnosing appendicitis and to evaluate site-related variations based on the frequency of US use. Additionally, we assessed the test characteristics of US when the appendix was clearly visualized.

Methods

Study Design

This was a secondary analysis of a prospective, observational study of children with suspected appendicitis.⁶ The Pediatric Emergency Medicine Collaborative Research Committee (PEM-CRC) reviewed and approved the final study protocol. Each participating site's institutional review board (IRB) also approved the study. Seven IRBs granted waivers of written informed consent or assent. At the three remaining sites, we obtained written consent from the guardians and assent from patients 7 years or older.

Study Setting and Population

This study was conducted at 10 PEDs located in tertiary care pediatric centers. All are members of the PEM-CRC of the American Academy of Pediatrics. We removed data from one site prior to analysis as its capture rate (proportion of eligible patients enrolled) was below 40%. A second site did not contribute to this secondary analysis as no study subject underwent US. The eight sites included were labeled A to H. We enrolled study subjects from March 2009 through April 2010.

Study Protocol

We enrolled children 3 to 18 years of age who presented to the ED with acute abdominal pain of <96 hours duration who were being evaluated for suspected appendicitis. “Suspected appendicitis” patients were defined as those for whom blood tests, radiologic studies (CT and/or US), and/or surgical consultation were obtained for the purpose of diagnosing appendicitis. We excluded patients with any of the following conditions: pregnancy, prior abdominal surgery (e.g., gastrostomy tube, abdominal hernia repair), chronic gastrointestinal illness or abdominal pain (e.g., inflammatory bowel disease, chronic pancreatitis, chronic or recurrent appendicitis), sickle cell anemia, cystic fibrosis, a medical condition affecting the provider's ability to obtain an accurate history (e.g., language or developmental delay), or history of abdominal trauma within 7 days of evaluation. Patients who had radiologic studies (CT or US) of the abdomen performed prior to emergency department (ED) arrival were also excluded. The detailed study procedures have been published previously, including those related to training of site staff, patient enrollment, data collection, and transmission to the central data management warehouse.²⁶

Clinicians obtained US at their discretion, not per study protocol. We abstracted data from the final US reports by the respective attending radiologists and determined whether appendicitis was present using key words that were defined a priori. We defined a positive US as the radiologist read of “appendicitis” or “perforated appendicitis” based on visualization of an abnormal appendix with or without secondary signs and a negative US as all other reported US results including “normal,” “appendix not visualized,” “equivocal,” and “other.”

Outcome Measures

Final diagnosis of appendicitis was determined by pathology, operative reports, or telephone follow-up. For those who underwent appendectomy, we determined the presence or absence of appendicitis by pathology reports. The presence or absence of perforation was determined from the attending surgeon's written operative report. In cases where a nonsurgical diagnosis was assigned, we contacted the family approximately 2 weeks after the ED visit to assess for resolution of signs and symptoms, visits to other sites of care, and need for surgery. For families who could not be contacted, we reviewed the medical record at the study institution to assess for any revisits within 3 months of the initial ED visit to determine if the patient underwent any abdominal imaging or operation at that facility.

Data Analysis

We performed descriptive analysis to characterize the study population. We calculated test characteristics of US (sensitivity, specificity, positive predictive value [PPV], negative predictive value [NPV], and likelihood ratios [LRs]) with 95% confidence intervals (CIs) for the entire cohort and for individual sites to assess site-related variations. Given that our data were obtained from several hospitals, the assumption of independent observations was not valid. We therefore used clustered sandwich standard error estimates to calculate the CIs. These standard errors allow for intrahospital correlation, relaxing the assumption that observations from the same hospital are independent. For the main analysis, we considered equivocal US results as test-negative. We performed a sensitivity analysis to assess the effect of both excluding and recategorizing equivocal cases as test-positive. Additionally, we assessed US test performance in the subset of cases where the appendix was clearly visualized.

Results

Of the 2,625 patients enrolled, 965 (36.8%) underwent abdominal US to evaluate for possible appendicitis and were analyzed for this study. We present the patient demographics, clinical characteristics, and outcomes in the overall cohort and in those for whom US was obtained in Table 1. US was performed at a median of 1.7 hours (interquartile range [IQR] = 0.9 to 2.8 hours) after physical examination. The median time from US examination to an operative procedure for those patients with presumptive diagnoses of appendicitis was 5.9 hours (IQR = 3.7 to 9.9 hours).

Table 1. Demographic and Clinical Characteristics.

Variable	US Cohort (N = 965)	Overall Cohort (N = 2,625)
Mean age, yr (±SD)	11 (±3.7)	10.8 (±3.8)
Male	433 (44.8)	1,338 (50.9)
Clinical findings <24 hours of abdominal pain	547 (56.6)	1,467 (55.9)
History of migration of pain to RLQ	384 (39.8)	1,080 (41.2)
Presence of maximal abdominal tenderness in RLQ	674 (69.8)	1,785 (68.0)
Rebound tenderness on exam	258 (26.7)	818 (31.2)
Abdominal guarding on exam	486 (50.3)	1,471 (56.1)
Increase in pain with walking	617 (64.0)	1,784 (68.0)
Appendicitis	324 (33.6)	1,018 (38.8)
Perforated appendicitis	74 (22.8)	275 (27.0)

Data are reported as n (%) unless otherwise noted

RLQ = right lower quadrant; US = ultrasound.

For all patients, US had a sensitivity of 72.5% (95% CI = 58.8% to 86.3%), specificity of 97.0% (95% CI = 96.2% to 97.9%), PPV of 92.5% (95% CI = 87.4% to 97.7%), NPV of 87.5% (95% CI = 84.3% to 90.7%), positive likelihood ratio (+LR) of 24.5 (95% CI = 15.6 to 38.3), negative likelihood ratio (–LR) of 0.28 (95% CI = 0.24 to 0.34), and overall accuracy of 88.8% (95% CI = 85.9% to 91.7%) for diagnosing appendicitis.

When we considered equivocal radiology reads as positive, sensitivity was 79.9% (95% CI = 64.4% to 95.5%), specificity 84.4% (95% CI = 76.4% to 92.4%), PPV 72.1% (95% CI = 62.5% to 81.8%), and NPV 89.3% (95% CI = 84.3% to 94.2%). Excluding the cases with equivocal radiology reads resulted in sensitivity of 78.3% (95% CI = 62.1% to 94.6%), specificity 96.6% (95% CI = 95.6% to 97.6%), PPV 92.5% (95% CI = 87.4% to 97.7%), and NPV 89.3% (95% CI = 84.3% to 94.2%).

We assessed the variation in test characteristics of US across sites based on the frequency of its use (Table 2). Sites A, B, and C used US as the first-line imaging modality 24 hours a day and used it for 89% to 94% of their enrolled patients. The sensitivity at these sites varied from 69% to 86%, with a combined sensitivity of 77.7% (95% CI = 54.7% to 99.9%). At site D, where US was available only during the day and used in 51% of subjects, its sensitivity was 51.6% (95% CI = 33.0% to 70.2%). The last four sites (E, F, G, and H) used US infrequently (9% of cases), and at these sites, the combined sensitivity was 35% (95% CI = 20.0% to 50.0%). The appendix was definitively identified in 56% of cases at sites A, B, and C combined; in 25% at site D; and in 26% at sites E, F, G, and H combined. Regardless of site, specificity of US varied little, between 96% and 99%.

Table 2. Variation in Use and Test Characteristics of US Across Sites.

Site, n	US Performed, n (%)	Pattern of Use of US/Availability-Pattern of Use	Prevalence, %	Sensitivity, % (95% CI)	Specificity, % (95% CI)	+LR (95% CI)	−LR (95% CI)
A, 291	259 (89)	24 hours a day/first-line test	44	86 (78–92)	97 (93–99)	30.9 (11.8–81.7)	0.14 (0.09–0.22)
B, 240	225 (94)	24 hours a day/first-line test	41	73 (63–82)	96 (91–99)	19.3 (8.1–46.0)	0.28 (0.19–0.39)
C, 269	249 (93)	24 hours a day/first-line test	26	69 (56–80)	96 (92–98)	18.2 (8.6–38.3)	0.32 (0.22–0.46)
D, 223	114 (51)	Daytime hours/used for 50% of patients	27	52 (33–69)	98 (91–100)	21.4 (5.2–87.8)	0.49 (0.34–0.71)
E, F, G, H, 1,308	118 (9)	Daytime hours/infrequent use	17	35 (25–50)	99 (94–100)	27.2 (8.5–86.9)	0.56 (0.43–0.72)

+LR = positive likelihood ratio; −LR = negative likelihood ratio; US = ultrasound.

When we restricted the analysis to those cases where the appendix was specifically identified and classified as normal or abnormal (469 cases, 48.6% of the total cohort), we found an overall sensitivity of 97.9% (95% CI = 95.2% to 99.9%), specificity 91.7% (95% CI = 86.7% to 96.7%), PPV 92.5% (95% CI = 87.4% to 97.7%), NPV 97.7% (95% CI = 94.7% to 99.9%), +LR 11.8 (95% CI = 7.7 to 18.2), and –LR 0.02 (95% CI = 0.009 to 0.05). Table 3 shows the variation in test characteristics of US across sites in this specific cohort. The sensitivity of US remained at least 94% at all sites.

Table 3. Variation in Use and Test Characteristics of US Across Sites: Analysis Restricted to Cases Where Appendix Was Visualized and Specifically Classified as Normal or Abnormal.

Site, N who had US	Pattern of Use of US-Availability/Pattern of Use	Prevalence of Appendicitis,%	Appendix Identified, n (%)	Sensitivity, % (95% CI)	Specificity, % (95% CI)	+LR (95% CI)	−LR (95% CI)
A, 259	24 hours a day/first-line test	52	190 (73)	100 (95–100)	96 (89–99)	22.8 (8.7–59.3)	0 (0-NaN)
B, 225	24 hours a day/first-line test	66	106 (47)	97 (89–100)	86 (70–95)	6.9 (3.1–15.8)	0.03 (0.008–0.13)
C, 249	24 hours a day/first-line test	41	114 (46)	96 (84–99)	90 (79–95)	9.2 (4.5–18.5)	0.05 (0.01–0.18)
D, 114	Daytime hours/used for 50% of patients	61	28 (25)	94 (69–100)	82 (48–97)	5.2 (1.5–18.2)	0.07 (0.01–0.5)
E,F,G, H, 118	Daytime hours/infrequent use	22	31 (26)	100 (56–100)	96 (77–100)	24.0 (3.5–163.5)	0.00 (0-NaN)

+LR = positive likelihood ratio; −LR = negative likelihood ratio; NaN = not a number; US = ultrasound.

Discussion

In this large multicenter study, we noted that use of US varied substantially among the participating children's hospitals; overall US sensitivity was low, but was generally higher at sites using it more frequently. Sites that used US as the primary imaging modality more frequently visualized the appendix. All sites achieved consistently high sensitivity when the appendix was clearly visualized.

Imaging with US or CT has become routine for most children undergoing diagnostic evaluation for appendicitis, with some believing that appendectomy should not be undertaken without imaging to confirm the clinical suspicion.¹⁰ A 2006 study found US of the appendix to increase diagnostic accuracy, alter management, and be more sensitive and specific than clinical impression, either alone or in conjunction with laboratory results.⁴ There is increasing interest in the use of US for appendicitis, but its use appears limited by concerns related to variable operator experience and overall perfor-mance.²³ As our study shows, centers with more experience with, and reliable access to, US for children use the modality more frequently. This is also reflected in the findings of a recent study that showed that even though the overall rate of use of CT in the United States for evaluation of abdominal pain in children increased every year from 1999 to 2007, the odds of undergoing CT were significantly lower among patients who presented to a pediatric-focused ED with access to US.²⁷

The overall sensitivity of US in our study is lower than prior estimates, although the specificity is similar. In a meta-analysis of studies published during the years 1986 to 2004, the authors found the pooled sensitivity and specificity of US for diagnosing appendicitis in children of 88% (95% CI = 86% to 90%) and 94% (95% CI = 92% to 95%), respectively.¹ In a 2010 systematic review, the authors reviewed studies published from January 2000 to March 2007 and found that the sensitivity of US for diagnosing appendicitis in children varied between 78 and 100% and the specificity from 88% to 98%.^5,15–21 More recent reports have noted sensitivity of 91% to 99% and specificity of 97% to 98%.^13,22 The overall lower sensitivity that we obtained in our study potentially provides a real-world estimate of test performance given the known interoperator variability in US conduct. As our study was conducted only at pediatric centers, US sensitivity may potentially be even lower at sites with less pediatric experience.

The variable sensitivity across sites appears in large part due to the varied ability to visualize the appendix. Visualization rates of the appendix with US vary widely in the published literature, from as high as 98% to as low as 22%.^15,24,28 Similar to other small studies, our data show that US accurately confirms or excludes appendicitis in cases where the appendix is clearly identified.^24,25 Our study supports the position of other authors that when US does not identify the appendix clearly, clinicians need to consider other modalities before diagnosing or excluding appendicitis.^23,29,30 These diagnostic modalities may include repeat clinical assessment; laboratory testing; admission for repeat clinical exams; and/or US, CT, magnetic resonance imaging, or discharge home with close follow-up.^31,32 Factors that influence visualization of the appendix, such as body mass index, need to be considered when ordering US and should be the focus of future studies that examine US utility.

Finally, the specificity of US was uniformly high at all centers, supporting the clinical policy statement of the American College of Emergency Physicians that US is, across institutions, more useful to confirm acute appendicitis rather than exclude it.⁵ Our results suggest the need for sites to implement quality improvement practices to track hospital-specific US performance. The threshold volume at which US performance improves warrants further investigation.

Limitations

Although we enrolled patients from numerous geographical regions, enrollment occurred exclusively in PEDs. US sensitivity may potentially be lower at community hospitals with less pediatric experience. Each site followed its own rather than a standard study protocol for US image acquisition and interpretation, which may have resulted in unaccounted-for differences across sites. Radiologists were not blinded to clinical team input or final outcomes. However, we used strict abstraction rules to assess the final radiology readings. We have no information about the experience of US technicians or radiologists involved in the conduct and interpretation of the studies; however, all the radiologists had pediatric training. Finally, patient-related factors such as age and body mass index, which can affect the test characteristics of US, were not assessed.

Conclusions

Ultrasound had an overall lower sensitivity in diagnosing appendicitis in children in this multicenter cohort than in previous reports. There was a large variation in rates of identification of the appendix and sensitivity for diagnosing appendicitis across sites, with lower rates at centers that used ultrasound less frequently. Ultrasound had high sensitivity and specificity, however, across all sites when the appendix was clearly identified. Other diagnostic modalities should be considered when US does not identify the appendix clearly.