Improving ultrasound for appendicitis through standardized reporting of secondary signs

Abstract

Objective

Our aim was to implement a standardized US report that included secondary signs of appendicitis (SS) to facilitate accurate diagnosis of appendicitis and decrease the use of computed tomography (CT) and admissions for observation.

Methods

A multidisciplinary team implemented a quality improvement (QI) intervention in the form of a standardized US report and provided stakeholders with monthly feedback. Outcomes including report compliance, CT use, and observation admissions were compared pre- and posttemplate.

Results

We identified 387 patients in the pre-template period and 483 patients in the posttemplate period. In the post-template period, the reporting of SS increased from 5.4% to 79.5% (p<0.001). Despite lower rates of appendix visualization (43.9% to 32.7%, p<0.001) with US, overall CT use (8.5% vs 7.0%, p=0.41) and the negative appendectomy rate remained stable (1.0% vs 1.0%, p=1.0). CT utilization for patients with an equivocal ultrasound and SS present decreased (36.4% vs 8.9%, p=0.002) and admissions for observations decreased (21.5% vs 15.3%, p=0.02). Test characteristics of RLQ US for appendicitis also improved in the posttemplate period.

Conclusion

A focused QI initiative led to high compliance rates of utilizing the standardized US report and resulted in lower CT use and fewer admissions for observation.

1.0 Introduction

1.1 Problem Description and Available Knowledge

Appendicitis remains the leading cause of pediatric abdominal pain requiring emergent surgery¹. Despite the prevalence of appendicitis, the clinical diagnosis remains challenging resulting in the use of diagnostic imaging. Ultrasound (US) of the right lower quadrant (RLQ) is recommended by the American College of Radiology and the American Academy of Pediatrics as the initial imaging modality in evaluating pediatric appendicitis^2,3. If the appendix is not visualized on US, then clinicians may doubt the US findings and utilize computed tomography scans (CT) or admissions for observation to assist in the diagnosis. CTs are an accurate diagnostic tool with reports of sensitivity (SN) ranging from 95 to 97% and specificity (SP) ranging from 94 to 97%⁴, but are more expensive than US and expose children to ionizing radiation, increasing their risk of subsequent cancer development^5-8. When the appendix is fully visualized, US can be as sensitive, specific, and accurate as CT^4,9-11; however, US is user-dependent as reflected by a wide appendix visualization rate ranging from 40% to 89%^11-15.

1.2 Problem

Absolute indications for subsequent imaging or admission for observation have not been clearly defined. Wide practice variation exists within and between children’s hospitals and non-children’s hospitals, resulting in inconsistent costs and resource utilization^16-18. Radiologists commonly record US findings in a free-handed US report. If the appendix is not visualized, then the impression is often a re-statement of the non-visualization of the appendix and suggestion of clinical correlation. The impressions of the equivocal US studies are thought to lack diagnostic information, so physicians then ordered CT, admission for observation, or both.

In an effort to increase the diagnostic accuracy of US, investigators have proposed combining equivocal US studies with additional data such as secondary signs (SS) of appendicitis^15,19-22. SS are sonographic descriptions of inflammation surrounding the appendix and include fluid collections, free fluid, echogenic fat, hyperemia, abnormal lymph nodes, abnormal adjacent bowel, bowel wall edema, and appendicoliths^19,23.

1.3 Rationale

Standardized reporting of US findings has been suggested as a means to provide clinicians with as much information as possible of many sonographic details and could assist in diagnosis even when the appendix is not fully visualized^{19,20,22,24,25}

1.4 Specific Aim

In order to optimize utility of US at our institution, we implemented a quality improvement (QI) initiative to increase the reporting of SS in RLQ US. Concurrently, we tracked the number of patients undergoing CT and the number of patients being admitted for observation.

2.0 Methods

2.1 Context

The QI effort took place at the Egleston Campus of the Children’s Healthcare of Atlanta (Atlanta, GA), a free-standing, university-affiliated, tertiary care pediatric hospital where more than 300 appendectomies are performed annually. The hospital serves children of all ages; however, we limited our study to children 5 to 18 years old. Age limits were used in concordance with ongoing efforts to utilize imaging for appendicitis diagnosis after use of a Pediatric Appendicitis Score (PAS) which requires patients to verbally describe symptoms. The emergency department, radiology, and surgical services are staffed by pediatric specialized attendings, staff, and trainees.

2.2 Intervention

Practice in our center is to have patients with concern for appendicitis assessed by emergency medicine physicians who determine the initial workup such as imaging studies. All US were performed by a radiology technician. Before our QI intervention, the US reports were dictated in an unstructured fashion by radiologists. When the appendix was not visualized, the impression often re-stated non-visualization of the appendix and recommended clinical correlation. This process resulted in follow up imaging in the form of CT, admission to the surgical service for observation, or both. Prior work from our group and others demonstrated that US reports that include details such as the presence or absence of specific SS may provide clinicians with reliable information even in the setting of a non- or partially visualized appendix^{19,20,22,24,25}. A multidisciplinary team of pediatric emergency medicine physicians, pediatric radiologists, pediatric surgeons, nurses, and QI personnel instituted a QI intervention to standardize the reporting of SS on US and to decrease the proportion of patients undergoing CT and being admitted for observation.

An aim to reduce CT use by 50% for patients with equivocal US was established for a 6-month time-frame (post-template period) with an additional 6 months of observation (sustainability period). During the post-template period, the multidisciplinary QI team met monthly to assess the use of the standardized report as well as address any specific concerns that were limiting the use of the template. During the sustainability period, formal meetings took place quarterly. Key drivers focused on standardization. A standardized US report template was adopted and uploaded to the electronic medical record reporting system²⁰.

2.3 Study of the Intervention

The success of the implementation of the standardized report was assessed by the increase in compliance of radiologists using the US report over time and the relative decrease in the proportion of patients with equivocal US studies undergoing CT or being admitted for observation. We defined compliance both as all seven SS mentioned and at least 5 of the 7 SS mentioned as we wanted acknowledge improved reporting even if it was imperfect. The study was a retrospective analysis of children (5–18 years old) with concern for appendicitis who underwent RLQ US from January 1, 2014 to December 31, 2015. We initiated a standardized US report that included appendix measurements, categorization of the appendix, and seven SS on January 1, 2015. To ensure generalizability, we aimed for our cohort to be as inclusive as possible. We used language recognition software to examine the chief complaints as listed in the electronic medical record and included all patients with chief complaints that included the terms: “abd,” “appy,” “stomach,” “appendicitis,” and “rlq.” Of these patients that we identified as having concern for appendicitis, we included all patients that received a RLQ US in order to evaluate the appendix. Patients were excluded if they underwent an US or CT for their abdominal pain at an outside hospital, if they had a prior appendectomy, if they were already being non-operatively managed for perforated appendicitis, or if they did not have abdominal pain. To ensure the integrity of the data, two reviewers (AP, KP) abstracted data from charts, and all final data were reviewed for accuracy by a single reviewer (KP).

The outcomes of interest were captured by the electronic medical record, and each admission note was reviewed to determine the clinical indication for admission. Final US reports were reviewed for primary and secondary signs of appendicitis. The primary sign of appendicitis was a fully visualized appendix with a diameter greater than or equal to 6mm¹⁹. SS included fluid collections consistent with abscesses (fluid collections), a significant amount of abdominal free fluid (free fluid), hyperechogenicity of periappendiceal fat (echogenic fat), increased regional bowel vascularity (hyperemia), the presence of enlarged or supranumery mesenteric lymph nodes (abnormal lymph nodes), hypoperistalsis or dilation of adjacent bowel loops (abnormal adjacent bowel), bowel wall edema, and appendicoliths^19,23. As has been previously described, US reports were classified into four categories: 1. Normal; 2. Equivocal without SS; 3. Equivocal with SS; and 4. Appendicitis^19,20,26. Categories 1 and 4 included a fully visualized appendix and were collectively referred to as unequivocal. Categories 2 and 3 included US in which the appendix was not fully visualized and were collectively referred to as equivocal. The final diagnosis of each patient was recorded as either appendicitis or not appendicitis. Appendicitis was confirmed through review of operative reports, pathology results, and CT impressions when CT was performed. Each patient’s electronic medical record was examined for details regarding the clinical course and any re-admissions. For patients diagnosed as not having appendicitis, chart review ensured appendicitis was not diagnosed in the 30 days after the initial presentation.

2.4 Intervention Implementation

Implementation of the standardized US report began with personal communication of the successful implementation of similar programs at other children’s hospitals²⁰. In order to shift the culture at our institution, several retrospective reviews were performed to validate the need to include SS in the reports and to educate clinicians regarding the reliability of SS as important variables to consider in making a diagnosis of appendicitis. The first assessed which SS were most highly associated with appendicitis²². The second demonstrated that SS were associated with duration of symptoms. Results of these studies were shared in local forums, QI meetings, and national meetings. Radiology champions (JL, KB), who were involved from the project start, facilitated consensus regarding the specific elements included in the final report, education for all radiology staff members, and dissemination of the templates in electronic form for ease of use. Initial iterations of the standardized report included SS. Subsequent versions also included a final classification into one of four categories as previously outlined. Figure 2 provides a p-chart demonstrating trends over time.

Figure 2.

Control Chart (p-chart) of monthly computer tomography scans (CTs) utilization rate and admission rate with standardized ultrasound report compliance rate.

2.5 Measures

Statistical process control (SPC) is a standard methodology for quality analysis and improvement and is typically reported on a standard SPC chart. SPC charts have a y-axis with measurements of improvement or decline in quality data plotted over time. We utilized control charts (p-charts) of the statistical process control (SPC) methodology to track our study’s quality data, which included report compliance, admission rates in all patients, and CT utilization rates in patients with an equivocal ultrasound with SS present. In Figure 2, the central lines demonstrate the mean rates for each of the three stages (pre-template period, post-template period, sustainability period). The pre-determined control limits are determined by a standard formula developed to reflect the capability of the process. The standard formula for control limits is $\overline{\mathit{p}}\pm 3\surd \overline{\mathit{p}}(1-\overline{\mathit{p}})∕n$ where $\overline{\mathit{p}}$ is the mean of base period and n is the number of observations during the specific length of time observed. We divided the study period into three stages based on the implementation practices: (1) pre-template stage (January 2014 to December 2014), (2) post-template stage (January 2015 to July 2015), and (3) sustainability stage (August 2015 to December 2015). We graphed the numbers of observations of the three quality data points on a monthly basis. The control limits that were calculated as a negative percentage were simply set to 0%, as is customary. Similarly control limits that were calculated as greater than 100% were set to 100%.

2.5 Analysis

Statistical analyzes were performed using SAS 9.4 (Cary, NC). Descriptive statistics were calculated for all variables of interest and included means and standard deviations, median and interquartile range, or frequencies and percentages, as appropriate. Statistical significance was assessed at the 0.05 level. Equivocal and unequivocal US patient demographics were compared using Chi-square and t-tests where applicable. Category 3 and 4 US reports were classified as positive US, and category 1 and 2 US reports were classified as negative US. Diagnostic test characteristics, including sensitivity, specificity, predictive values, and accuracy, were calculated.

2.5 Ethical Considerations

The standardized report was aimed to improve the outcomes of all patients with suspected appendicitis; however, it was the responsibility of the individual radiologist to comply. The standardized reports were considered “value-added” with negligible risk of adverse effects. The use of the report did not limit a physician’s ability to order a subsequent CT or admission if he or she determined these were clinically necessary. Approval was obtained from the Emory University Internal Review Board (#00077519) before the retrospective study.

3.0 Theory

We hypothesized that standardization of RLQ US reports would increase the number of SS reported and increase the accuracy of US. We believed the improved utilization of US would decrease CT use and admissions for observation.

4.0 Results

4.1 Process Measures and Outcomes

We identified 387 patients in the pre-template period and 483 patients in the post-template period. There was no difference in race, gender, or age between the pre- and post-template period. Overall CT use (33/387(8.5%) vs 34/483(7.0%), p=0.413) and the negative appendectomy rate remained low (4/387(1.0%) vs (5/483)1.0%, p=1.0). Prior to the implementation of the report template, only 4/387(1.0%) of free-handed US reports mentioned all seven SS and only 21/387(5.4%) mentioned at least 5 of the 7 SS. In the post-template period, the reporting of at least 5 of the 7 SS (384/483(79.5%), p<0.001) and the reporting of all seven SS (333/483(69.8%), p<0.001) significantly increased. CT utilization for patients with an equivocal ultrasound and SS present decreased (16/44(36.4%) vs 4/45(8.9%), p=0.002). Overall admissions for observations decreased (83/387(21.5%) vs 74/483(15.3%), p=0.020).

4.2 Observations of the Intervention

Test characteristics of US as a diagnostic test improved in the post-template period. We defined equivocal US with no SS as a negative US for appendicitis, and equivocal US with SS present as a positive US for appendicitis. The final pathology as transmural inflammation was used as our gold standard. With the implementation of the standardized US template, sensitivity, specificity, and accuracy increased. The negative predictive value increased and positive predictive value decreased in 2015 as compared to 2014 as expected given the decrease in the prevalence of appendicitis. When the US diagnostic result categories are stratified by gender, we observed that males are more likely to be a true positive US than females (33.1% vs 16.8%, p<0.001); however, males were more likely to have appendicitis in general. There were no differences in false positive US or false negative US rates between genders. Using the presence of SS on US, the diagnosis of appendicitis can be made equally as well in males and females (84.9% vs 87.5% accuracy). When the US diagnostic result categories are stratified by race, there is no difference between races.

4.3 Consequences of the Intervention

Our overall appendix visualization rate of 37.7% was lower than we initially anticipated but is similar to recently published rates 12, 15, 23. In the post-template period, we observed a decrease in appendix visualization (43.9% to 32.7%, p<0.001) and a concomitant decreased rate of appendicitis (35.7% vs 26.5%, p=.004). We hypothesize that these results are secondary to the increased US usage at our institution over time for patients that presented with right lower quadrant pain but did not have final diagnoses as appendicitis as providers began to trust in the diagnostic accuracy of US. The decreased appendix visualization rate is expected with a decreased rate of appendicitis since normal appendices are more difficult to visualize on US than inflamed, dilated appendices. We reviewed the imaging of patients presenting with appendicitis to confirm that ultrasound use had not increased in this group. The total number of appendectomies performed on patients with appendicitis at least five years old in 2014 and 2015 was 638. Of the 638, 217 presented with outside imaging. Of the 427 patients that presented to our ED without imaging, 109 patients did not undergo pre-operative imaging, 308 had US as initial imaging, and 10 had CT as their initial imaging. In 2014, 152/211 (72%) patients that presented to our ED without imaging and underwent an ultrasound as the initial imaging study prior to appendectomy. In 2015, 157/216 (73%) patients that presented to our ED without imaging and underwent an ultrasound as the initial imaging study prior to appendectomy. The overall use of US imaging for patients that underwent an appendectomy did not differ between 2014 and 2015; therefore, observed increase in the use of ultrasound was specifically for children presenting with right lower quadrant pain but did not have a final diagnosis of appendicitis.

5.0 Discussion

5.1 Summary

There is potential to improve the diagnostic accuracy of US for pediatric appendicitis, as well as, patient outcomes using standardized reports that include details such as SS. Our study demonstrates that implementation of a standardized US reports increased reporting of SS, decreased CT use, and decreased admissions for observation in patients with suspected appendicitis and equivocal US. This study was conducted in a high-volume center and can easily be emulated by others interested in optimizing use of US while minimizing resource utilization.

5.2 Interpretation

Traditionally, non-visualization of the appendix resulted in an equivocal imaging report and a diagnostic dilemma. Among children with an equivocal US, the lack of diagnosis results either in delay in diagnosis or overutilization of resources. Prior studies have demonstrated that a standardized report decreased CT use; however, our study is the first to demonstrate a decrease in admissions for observation²⁰. After implementation of the standardized US report, the overall CT use and the negative appendectomy rate remained low (33/387(8.5%) vs 34/483(7.0%), p=0.413; 4/387(1.0%) vs (5/483)1.0%, p=1.0)). Patients with equivocal US and SS present traditionally represent the most difficult patients to diagnose^20,22. Based on our prior work we recommend that patients with hyperemia, fluid collections, an appendicolith, or any combination of two SS proceed to appendectomy without undergoing CT or observation²².

Our overall appendix visualization rate of 328/870(37.7%) was lower than we initially anticipated but is similar to recently published rates^12,15,23. In the post-template period, we observed a decrease in appendix visualization (170/387(43.9%) to 158/483(32.7%), p<0.001) and a concomitant decreased rate of appendicitis (138/387(35.7%) vs 128/483(26.5%), p=.004). This direct correlation is expected since normal appendices are more difficult to visualize on US than inflamed, dilated appendices. As US became more reliable, clinicians may have begun to liberalize the use of US for lower PAS patients and thus made the visualization rate lower. Though there is a statically significant difference in the PAS scores between the pre-template and post-template periods (mean ± sd: 6.2 ± 1.9 vs 6.7 ± 1.9, p=0.001), there is no clinical difference between the two years. The numerical values of both years highlight that children with a median PAS are the hardest to diagnose by a clinical score and require diagnostic imaging. Notably, there were no guidelines regarding the PAS and US result, as the ED physicians independently obtained PAS and RLQ US as they saw clinically relevant.

The interpretation of an US as positive or negative was strictly based on the visualization of the appendix, measurement of the size of the appendix, and secondary signs present if the appendix was not visualized. On subsequent version of the template, the radiologist’s final diagnosis was added but we did not change the interpretation of the US based on the radiologist’s impression, since this information was not available for a large portion of patients. Given this, there are some instances when an appendix measured greater than 6mm but the radiologist’s impression was not appendicitis due to lack of secondary signs or data that a larger appendiceal diameter is more accurate for diagnosing appendicitis. We recognize that there are several different appendiceal diameters that could be used as the positive test criteria for appendicitis^27,28. We selected greater than or equal to 6mm as an abnormal appendiceal size, since this is the most traditional definition of appendicitis; however, 7mm is the typical measurement of abnormal appendix at our institution. We did a post hoc analysis of the data using different criteria as a positive test criterion including 6mm with SS, 7mm and 8mm in addition to the original 6mm. As expected as the appendix diameter requirement increased in size, the sensitivity decreased and the specificity increased. The test characteristics of US when applying the four test criteria (6mm, 6mm with SS, 7mm, and 8mm) were similar. For all four criteria, the negative predictive value remained significantly different between the pre- and post-template periods. Notably, the only other test characteristic that was significantly different between pre- and post-template periods, was the accuracy when 6mm with SS was used as the positive test criterion (88.4% vs 83.2%, p=0.028). The accuracy of all using different maximal appendix diameters was similar in the post-template period: 6mm (87.6%), 6mm with SS (88.4%), 7mm (88.5%), and 8mm (88.0%), so the use of any of these is a reasonable choice at our institution.

There were no issues regarding cost of implementation, failures of implementation, or missing data. All patients that underwent a RLQ US had an accompanying US report. Since we were studying the mention or lack of mention of particular secondary signs, there were no missing data points for the presence or absence of secondary signs mentioned. We do recognize that the fact that a sign was not mentioned, does not necessarily guarantee that it was in fact to present on the original US. In our prior manuscript, we had radiologists retrospectively review the original US in order to validate that if a free-handed report did not mention a SS, then it was in fact not present. 93% of SS not mentioned were not present on the original US on subsequent review²².

5.3 Limitations

Our study includes several limitations including the fact that this work is a retrospective analysis of data collected from a single hospital system. Interestingly, our results may be generalizable due to our diverse patient population since we are a tertiary referral center. Though retrospective, we carefully selected a cohort in which there was a high clinical suspicion for appendicitis by thoroughly examining emergency department notes and manually abstracting details from operative and pathology reports. For patients who were deemed not to have appendicitis, we tracked their clinical course to ensure they were not readmitted after discharge and found two patients that were re-admitted and diagnosed with appendicitis. Interestingly, these two readmissions only occurred in the pre-template period. Based on our status as the pediatric referral center, patients were expected to return to our system if medical care was needed. Our study did not include the patients who had clinical presentations that were so highly suggestive of appendicitis that they did not undergo preoperative imaging, so our patients represent those whose diagnosis was challenging and required diagnostic imaging. Despite this being a retrospective review, we attempted to ensure highly accurate data collection by using two reviewers to abstract the data (KP, AP) and used two radiologists to review imaging (KB, JL). We were impressed by the ready implementation of the standardized report usage on the part of the Radiology Department. Having two champions and departmental leadership support of this endeavor facilitated success.

6.0 Conclusions

We implemented a focused QI initiative to incorporate a standardized US report for all patients undergoing RLQ US for diagnosis of appendicitis in a tertiary pediatric hospital setting. Rapid incorporation of the standardized report was facilitated by close collaboration with colleagues in the Radiology. As intervention compliance improved, CT use and admissions for observation among patients with equivocal US results decreased demonstrating improved resource utilization; however, we did observe an increase in the rate of ultrasonography for patients that did not have appendicitis. We recommend an accompanying quality improvement project in the judicious use of ultrasonography if the goal is to decrease cost. Our institution’s CT rate was low prior to the implementation of a standardized report. The implementation of a standardized US report would be expected to be even more transformative in pediatric hospitals with a high CT utilization rate.

Figure 1.

Diagram of the specific aim and description of the key drivers of change for instituting a surgical quality improvement project to decrease the number of computer tomography scans (CTs) and admissions after an equivocal right lower quadrant (RLQ) ultrasound (US) for appendicitis.

Table 1.

Demographics, imaging, and diagnosis of patients with high clinical suspicion for appendicitis based on equivocal or unequivocal right lower quadrant ultrasounds.

Characteristic	Overall (N = 870) N (%)	2014 (N = 387) N (%)	2015 (N = 483) N (%)	p-value
Race
White	404 (46)	191 (49)	213 (44)	0.103
Black	370 (43)	162 (42)	208 (43)
Other	96 (11)	34 (9)	62(13)
Sex
Female	447 (51)	196 (51)	251 (52)	0.699
Male	423 (49)	191 (49)	232 (48)
Age (years), mean ± sd	11.1 ± 3.7	11.0 ± 3.6	11.1 ± 3.8	0.761
PAS Score, mean ± sd (N = 564)	6.4 ± 1.9	6.7 ± 1.9	6.1 ± 1.9	0.001*
Imaging Pathway (N = 870)
Ultrasound with CT	67 (8)	33 (9)	34 (7)	0.413
Ultrasound only	803 (92)	354 (92)	449 (93)
Ultrasound visualization (N = 870)
Appendicitis	195 (22)	94 (24)	101 (21)	0.001*
Equivocal with SS	89 (10)	44 (11)	45 (9)
Equivocal without SS	453 (52)	173 (45)	280 (58)
No Appendicitis	133 (15)	76 (20)	57 (12)
Final Diagnosis
Appendicitis	266 (31)	138 (36)	128 (27)	0.004*
No Appendicitis	604 (69)	249 (64)	355 (74)

^*Indicates statistical significance, sd=standard deviation, PAS=pediatric appendicitis score, CT=computed tomography, SS=secondary signs

Table 2.

Comparison of test characteristics of ultrasound between the pre- and post-template period.

	2014	2015	p-value
Sensitivity	78.3% (70.4 – 84.8)	83.6% (76.0 – 89.6)	0.271
Specificity	88.0% (83.3 – 91.7)	89.0% (85.3 – 92.1)	0.689
PPV	78.3% (70.4 – 84.8)	73.3% (65.3 – 80.3)	0.327
NPV	88.0% (83.3 – 91.7)	93.8% (90.6 – 96.1)	0.014*
Accuracy	84.5% (80.5 – 88.0)	87.6 (84.3 – 90.4)	0.186

^*95% CI in brackets, Indicates statistical significance, PPV=positive predictive value, NPV=negative predictive value, CI=confidence intervals

Abbreviations

CT

computed tomography scans

PAS

pediatric appendicitis score

QI

quality improvement

RLQ

right lower quadrant

SN

sensitivity

SP

specificity

SS

secondary signs of appendicitis

US

ultrasound