Abstract
OBJECTIVE
The purpose of this study was to assess radiologists’ adherence to published guidelines for managing renal masses detected at abdominal CT at one institution and to a critical results communication policy.
MATERIALS AND METHODS
A validated natural language processing tool supplemented by manual review was used to randomly assemble a cohort of 97 radiology reports from all abdominal CT reports (n = 11,952) generated from July 2010 to June 2011. Critical renal mass findings warranted consideration for surgery, intervention, or imaging follow-up and required direct, separate, and timely communication to the referrer in addition to the radiology report. Primary outcomes were adherence to guidelines and institutional policy for communicating critical results. Sample size allowed a 95% CI ± 5% for primary outcome. Pearson chi-square test was performed to assess whether radiology subspecialization was predictive of the primary outcome.
RESULTS
Of all abdominal CT reports, 35.6% contained at least one renal mass finding (4.3% critical). Guideline adherence was lower for patients with critical than for those with noncritical findings (48/57 [84.2%] vs 40/40 [100%]; p = 0.01). Adherence to critical result communication policy was 73.7% (42/57). For critical findings, abdominal radiologists had higher guideline adherence (40/43 [93.0%] vs 8/14 [57.1%]; p = 0.001) and critical result communication policy adherence (36/43 [83.7%] vs 6/14 [42.9%]; p = 0.002) than non-abdominal radiologists.
CONCLUSION
In reporting renal masses detected at abdominal CT, radiologists largely adhered to management guidelines but did not adhere to the critical results communication policy in one of four reports. Subspecialization improved adherence to both management guidelines and the institution's critical result communication policy.
Keywords: abdominal CT, American Urological Association guidelines, critical test results, evidence-based practice, renal cyst, renal mass
Renal masses are commonly found on abdominal CT scans. It is estimated that 50% of adults older than 50 years have at least one cystic or solid renal mass [1–3]. With the advent of high-spatial-resolution CT scanners, it is likely that renal masses are being diagnosed in more patients [4]. Most renal masses are detected incidentally, and their management is controversial [5]. Renal cell carcinoma is curable only when confined to the organ, so early detection and treatment are important [6]. However, most renal masses are benign, most being simple cysts, and their discovery may engender unnecessary treatments and unnecessary, costly, and sometimes morbid diagnostic tests [7]. Therefore, radiologists’ recommendations are important because management is often based solely on the imaging findings. Failure to provide appropriate recommendations could lead to not treating cancer at a curable stage or to unnecessarily treating benign findings. Published practice guidelines include recommendations for the management of renal masses detected at abdominal CT [7–10]. In general, indeterminate cystic masses and solid masses (except those that can be diagnosed as small angiomyolipomas) warrant consideration for surgery, intervention, or imaging follow-up [7–9].
Communication of critical results is also important to ensure safe patient care and positive outcomes. In addition, failure to communicate test results is one of the most common reasons for medical malpractice lawsuits [11–13]. Failure to document such communication is one of the most common reasons radiologists lose lawsuits [14]. Therefore, assuring communication of critical results has been a National Patient Safety Standard of The Joint Commission since 2005 [15]. At our institution, a critical result communication policy considers renal masses that warrant consideration for surgery, intervention, or imaging follow-up of critical findings when they are unexpected or newly detected. Therefore, according to our policy, a special alert, that is, a direct, separate, and timely communication, to referring physicians is required in addition to the standard radiology report [16]. Under the policy, renal masses that are reliably considered benign, expected, or already known are not considered critical and do not require additional communication beyond the radiology report.
Studies have addressed radiology reporting limitations and techniques for improving communication of radiology test results [17–21]. However, we are unaware of any investigations assessing whether radiologists’ follow-up recommendations adhere to guidelines for the management of renal masses or how radiologists communicate renal mass findings, particularly critical findings. Our objective in this study was to assess radiologists’ adherence to published guidelines for managing renal masses detected at abdominal CT at our institution and to our institution's policy for communicating critical results for such masses.
Materials and Methods
Study Setting and Cohort
Institutional review board approval was obtained for this retrospective HIPAA-compliant study, performed at a 793-bed urban university-affiliated tertiary care hospital with 3.5 million annual ambulatory visits and more than 500,000 radiologic examinations annually. We assembled the study cohort of abdominal CT reports using a randomization protocol detailed later. The institution's radiology information system (IDX Rad v9.2, GE Healthcare) was used to identify all abdominal CT scans (n = 11,952) obtained between July 1, 2010, and June 30, 2011 (Fig. 1). From these reports, a publicly available and previously validated natural language processing (NLP) tool [22] was used to retrieve radiology reports that mentioned detection of a renal mass. The NLP tool searched the findings and impression sections of the reports for terms (and their plural forms) that included “mass,” “nodule,” “hypodensity,” “hyperdensity,” “hypodense,” “hyperdense,” “nodularity,” “lesion,” “cystic,” and “cyst” combined with “renal” or “kidney” in the same sentence. We used a rule-based negative expression algorithm (NegEx) [23] to exclude reports with negated findings, such as “no kidney masses.”
The NLP tool identified 4558 (38.1%) reports with at least one renal mass. From these, a single investigator (a radiologist in the second year of fellowship) manually reviewed 500 randomly selected reports to identify reports containing findings considered critical, a sample size selected to obtain estimates of adherence both to published guideline recommendations and to our institution's critical result communication policy with a 95% CI not wider than 5%. During manual review, 32 reports were excluded because they did not include a renal mass finding. Because 468 of the 500 reports described a renal mass, the positive predictive value for the NLP tool was 93.6% (95% CI, 91.5–95.7%).
Of 468 reports with renal mass findings, 57 (12.2%) included renal mass findings that were considered critical according to our institution's definition of a critical test result, that is, a new or unexpected finding that generally warrants consideration for surgery, intervention, or imaging follow-up. The other 411 reports included renal mass findings that were not considered critical. The final study cohort of 97 abdominal CT reports included all 57 reports with critical findings and 40 randomly selected from the 411 reports without a critical finding. By use of an alpha value of 0.05 and an assumption of 95% adherence in the critical finding group, a sample size of 96 was required to detect a 20% difference from the comparison group. These reports were derived from CT scans of 97 different patients (42 [43.3%] women, 55 [56.7%] men; age range, 19–89 years; mean, 63.7 years). A total of 63 reports derived from abdominal radiologists and 34 from other radiologists. The 57 reports with critical renal mass findings were generated by 16 radiologists (six nonabdominal and 10 abdominal subspecialists). The 40 reports with noncritical renal mass findings were generated by 18 radiologists (seven nonabdominal and 11 abdominal sub-specialists). From this cohort of 97 patients with renal mass findings, 75 (77.3%) of the CT scans were obtained only after IV administration of contrast material. Of these, 40 (53.3%) had critical renal mass findings, and 35 (46.7%) had noncritical renal mass findings. Among the 97 patients, 17 (17.5%) of the CT scans were obtained both with and without IV contrast material. Of these, 12 (70.6%) had critical mass findings and five (29.4%) had noncritical renal mass findings. Finally, five (5.2%) patients underwent unenhanced CT. All had critical renal mass findings.
Each of the 97 reports was independently reviewed by two experienced abdominal radiologists (18 and 22 years’ experience) using the definitions outlined later to determine whether the reports were adherent to published guidelines regarding management recommendations for renal masses detected at abdominal CT and our institution's policy for communication of critical results for such masses. Disagreements were resolved by consensus during an in-person meeting.
Defining Renal Mass Management
Recommendations and Critical Results
Publications used to define radiologist adherence to renal mass management guidelines included a comprehensive review in Radiology [7], guidelines of the American Urological Association [9], American College of Radiology (ACR) renal mass evaluation appropriateness criteria [24], and an ACR white paper on managing incidental findings at abdominal CT [8]. Although the white paper was published after our study start date, it reinforced the Radiology publication and focused on management. We amalgamated the content of these publications to summarize the recommendations and to relate them to our institution's definition of a critical result (Fig. 2). Based on the well-established Bosniak classification described in the ACR appropriateness criteria and ACR white paper, cystic renal mass categories I and II were considered benign, not warranting follow-up and therefore were not considered critical results. Bosniak IIF, III, and IV cystic renal masses warrant consideration for imaging follow-up, surgery, or intervention and were considered critical results [6, 7, 9, 24–26] if the finding was new or unexpected. We defined solid renal masses as those described as solid or predominantly enhancing. All renal masses reported as solid, except angiomyolipomas (noncalcified renal masses containing fat), also generally warrant consideration for surgery, intervention, or imaging follow-up and were considered critical results [9, 25, 26] if new or unexpected. Renal masses stable in follow-up studies and reported as critical results in a previous report were not considered critical. For masses detected with unenhanced CT, findings were considered critical if the radiologist recommended surgery, intervention, or additional imaging.
Recommendations regarding treatment or imaging follow-up of angiomyolipomas were also included. Because of risk of spontaneous bleeding, angiomyolipomas 4 cm or larger were considered critical results [27, 28].
Institutional Process for Communicating Critical Results
In February 2006, a quality improvement initiative was implemented at our institution that required radiologists to communicate critical findings to referring providers and document the communication in the radiology report [29]. In patients with critical findings, an institution policy–adherent radiology report included language that documented a closed-loop form of communication, that is, that the communication was received. Initially, communications of critical results to referrers or associated health care providers, particularly critical results that were urgent, were limited to direct, either face to face or by telephone call. In December 2009, our institution implemented an automated, electronic alert notification system that also allowed HIPAA-compliant electronic notifications of nonurgent critical results and a means of following up and documenting that all critical result communications (both urgent and nonurgent) were received [30].
Outcome Measures
Primary outcome measures were how often radiologists adhered to published guidelines for renal mass findings and how often they adhered to our institution's policy for communicating critical results regarding renal masses detected at abdominal CT. Reports were considered nonadherent when a critical result was encountered and a recommendation was not provided or when a recommendation was inconsistent with the guidelines. When more than one renal mass was encountered, the report was considered non-adherent if the recommendations for any one mass were inconsistent with the guidelines. For patient reports of both critical and noncritical findings, the critical finding was analyzed. Secondary outcome measures were whether there were significant differences in the proportion of reports that were adherent to the recommendation guidelines or the critical result communication policy between those derived from abdominal radiologists (abdominal radiology division) and those derived from other radiologists (emergency, oncoradiology, and community divisions in the department).
Statistical Analysis
The Z test was applied to calculate 95% CIs for proportion of reports adherent to published recommendations and to our institutional communication of critical results policy. Comparison of the proportion of abdominal radiologists’ reports adherent to recommendations or to the critical result communication policy relative to other radiologists’ reports was performed by Pearson chi-square test with p < 0.05 considered significant. Statistical analyses were performed with SPSS software (version 15, IBM SPSS).
Results
Prevalence of Renal Masses and Renal Mass Critical Findings
The NLP tool had initially identified 4558 of 11,952 (38.1%) abdominal CT reports that mentioned at least one renal mass. However, manual review excluded 32 of the 500 randomly selected manually reviewed reports because they contained no renal mass findings. Thus in total, 35.6% of all abdominal CT reports mentioned at least one renal mass (468/500 of 4558/11,952). Of all 468 renal mass findings, 57 (12.2%) were critical. Therefore, 4.3% of all abdominal CT reports (57/468 of the 35.6% abdominal CT reports with at least one renal mass) contained a renal mass considered to be a critical finding. In addition, 31.3% of all abdominal CT reports contained a renal mass finding considered noncritical (35.6% – 4.3% = 31.3%).
Adherence to Guidelines and Critical Results Communication Policy
Of 57 reports with critical findings, 48 (84.2%) were adherent to guideline recommendations, and 42 (73.7%) were adherent to the critical result communication policy. Guideline adherence was significantly lower for reports with critical results 48/57 (84.2%) compared with reports without a critical result 40/40 (100%) (p = 0.01). Similarly, adherence to the critical result communication policy was significantly lower for reports with critical results 42/57 (73.7%) compared with reports without a critical result 40/40 (100%) (Fig. 3). Among reports with a critical result, guideline adherence was significantly higher for abdominal radiologists (40/43 [93.0%]) than other radiologists (8/14 [57.1%]) (p = 0.001) (Fig. 4A). Adherence to the critical result communication policy was also significantly higher for abdominal radiologists’ reports (36/43 [83.7%]) than those by other radiologists (6/14 [42.9%]) (p = 0.002) (Fig. 4B).
Discussion
Renal masses are ubiquitous. More than one third of abdominal CT scans in our study had reports mentioning at least one. Although most renal masses are benign and do not warrant a specific management action, some warrant consideration for surgery, intervention, or imaging follow-up. These we defined as critical results (if new or unexpected) and were found in 4.3% of all abdominal CT reports. Other renal masses are generally ignored, or at most added to the patient's problem list in the medical record should a clinical problem related to the finding arise in the future. Examples of these findings included benign cysts that could eventually cause pain, rupture, bleed, or become infected.
When considering all renal mass findings detected at abdominal CT, more than nine of 10 of our radiologists’ reports included recommendations that were commensurate with published guidelines. However, a significantly larger proportion of abdominal radiology subspecialists’ management recommendations were adherent than were other radiologists. Indeed, subspecialty radiologists might be expected to be more familiar with the literature pertinent to their subspecialty than radiologists who do not subspecialize. Moreover, nearly 16% of reports of critical renal mass findings did not adhere to published recommendation guidelines, and among the reports of nonabdominal radiologists, almost one half (43%) contained no recommendations or contained recommendations that were inconsistent with published guidelines. This has important clinical implications. Because renal mass management often is determined on the basis of imaging findings alone, referrers often rely on radiologists’ opinions as to what might be helpful diagnostically. Among renal masses for which the differential diagnosis includes benign and malignant conditions, the failure to include management recommendations in the radiology report, such as specific additional imaging tests and percutaneous biopsy, could result in a benign entity being inappropriately treated with surgery or ablation or a malignant one being inappropriately followed or ignored.
Use of closed-loop forms of communication, those that ensure the communication was received, is important for transmission of critical results. Failure to communicate findings or a delay in their communication may result in delays in implementation of recommendations or in recommendations that are not followed at all. More than 26% of reports of critical results found in the evaluation were not communicated as such. When a renal mass was identified as a critical result, a significantly greater fraction of abdominal radiologists’ reports were adherent to our institutional critical result communication policy than were other radiologists’ reports.
Our study had limitations. First, it was conducted at one tertiary care institution, and therefore the results may not be applicable to other health care settings. Second, the critical results communication policy evaluated in this study is specific to our institution. However, the definition of a critical result, description of our system of alerting referring physicians of critical results, and the evaluation of our radiologists’ adherence to our policy may allow other institutions to consider implementation of similar policies for their practices. Finally, we did not analyze differences in the proportion of critical renal mass findings among different CT protocols. This was not our purpose. It is already known that the type of protocol would be expected to affect the likelihood of a critical renal mass finding. CT scans obtained without IV contrast material or only with contrast material would be expected to yield more indeterminate renal masses compared with a CT examination performed with and without IV contrast material; these might warrant additional imaging. On the other hand, the latter more thorough protocol may yield more masses suspicious for cancer; these might warrant surgery or intervention. To assess radiologists’ adherence to published guidelines and our institution's critical result communication policy, we chose a cohort that would be expected to have patients who were examined with multiple protocols, because it is well known that renal masses are detected on all CT scans of the abdomen.
In summary, although most renal masses detected with abdominal CT are benign, a clinically important portion of them (more than 12% of reports with renal mass findings and 4.3% of all abdominal CT reports in our study) warranted consideration for surgery, intervention, or imaging follow-up. Although most of our radiologists’ management recommendations adhered to published guidelines for renal masses detected with abdominal CT, when critical results were considered, nearly 16% of the reports were not adherent. We also observed a significant difference in adherence between abdominal subspecialists and other radiologists. When only critical results were considered, nearly one half of nonabdominal radiologists’ reports contained no recommendations or contained recommendations that were inconsistent with guidelines.
Future studies would be useful to assess interventions to improve adherence to published guidelines or adherence to critical test result communication policies for renal mass findings detected with abdominal CT. Such studies are likely to be particularly useful for radiologists who do not subspecialize in abdominal radiology. It remains to be seen whether health information technology–enabled interventions using decision support at the time of reporting will be helpful in closing the current performance gaps. Because the management of renal masses is often predicated solely on imaging findings, the inclusion of appropriate recommendations in the radiology report and provision of closed-loop forms of communication will likely improve the care of patients with renal masses, resulting in prompt and proper treatment of renal cancer and the appropriate management of renal masses not fully diagnosed with CT.
Acknowledgments
C. K. Maehara supported in part by the National Institutes of Health, National Library of Medicine (Boston-area Research Training Program in Biomedical Informatics, grant T15LM007092). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Library of Medicine or the National Institutes of Health.
References
- 1.Luciani LG, Cestari R, Tallarigo C. Incidental renal cell carcinoma-age and stage characterization and clinical implications: study of 1092 patients (1982–1997). Urology 2000. 56:58–62. doi: 10.1016/s0090-4295(00)00534-3. [DOI] [PubMed] [Google Scholar]
- 2.Terada N, Ichioka K, Matsuta Y, Okubo K, Yoshimura K, Arai Y. The natural history of simple renal cysts. J Urol. 2002;167:21–23. [PubMed] [Google Scholar]
- 3.Tada S, Yamagishi J, Kobayashi H, Hata Y, Kobari T. The incidence of simple renal cyst by computed tomography. Clin Radiol. 1983;34:437–439. doi: 10.1016/s0009-9260(83)80238-4. [DOI] [PubMed] [Google Scholar]
- 4.Hollingsworth JM, Miller DC, Daignault S, Hollenbeck BK. Rising incidence of small renal masses: a need to reassess treatment effect. J Natl Cancer Inst. 2006;98:1331–1334. doi: 10.1093/jnci/djj362. [DOI] [PubMed] [Google Scholar]
- 5.Lindblad P. Epidemiology of renal cell carcinoma. Scand J Surg. 2004;93:88–96. doi: 10.1177/145749690409300202. [DOI] [PubMed] [Google Scholar]
- 6.Tsui KH, Shvarts O, Smith RB, Figlin RA, de-Kernion JB, Belldegrun A. Prognostic indicators for renal cell carcinoma: a multivariate analysis of 643 patients using the revised 1997 TNM staging criteria. J Urol 2000. 163:1090–1095. doi: 10.1016/s0022-5347(05)67699-9. [DOI] [PubMed] [Google Scholar]
- 7.Silverman SG, Israel GM, Herts BR, Richie JP. Management of the incidental renal mass. Radiology. 2008;249:16–31. doi: 10.1148/radiol.2491070783. [DOI] [PubMed] [Google Scholar]
- 8.Berland LL, Silverman SG, Gore RM, et al. Managing incidental findings on abdominal CT: white paper of the ACR incidental findings committee. J Am Coll Radiol. 2010;7:754–773. doi: 10.1016/j.jacr.2010.06.013. [DOI] [PubMed] [Google Scholar]
- 9.Campbell SC, Novick AC, Belldegrun A, et al. Practice Guidelines Committee of the American Urological Association. Guideline for management of the clinical T1 renal mass. J Urol. 2009;182:1271–1279. doi: 10.1016/j.juro.2009.07.004. [DOI] [PubMed] [Google Scholar]
- 10.Bluth EI, Bush WH, Jr, Amis ES, Jr, et al. Indeterminate renal masses: American College of Radiology—ACR Appropriateness Criteria. Radiology. 2000;215(suppl):747–752. [PubMed] [Google Scholar]
- 11.Berlin L. Communicating findings of radiologic examinations: whither goest the radiologist’s duty? AJR. 2002;178:809–815. doi: 10.2214/ajr.178.4.1780809. [DOI] [PubMed] [Google Scholar]
- 12.Berlin L. Standards, guidelines, and roses. AJR. 2003;181:945–950. doi: 10.2214/ajr.181.4.1810945. [DOI] [PubMed] [Google Scholar]
- 13.Johnson CD, Krecke KN, Miranda R, Roberts CC, Denham C. Quality initiatives: developing a radiology quality and safety program: a primer. RadioGraphics. 2009;29:951–959. doi: 10.1148/rg.294095006. [DOI] [PubMed] [Google Scholar]
- 14.Raskin MM. Survival strategies for radiology: some practical tips on how to reduce the risk of being sued and losing. J Am Coll Radiol. 2006;3:689–693. doi: 10.1016/j.jacr.2006.03.018. [DOI] [PubMed] [Google Scholar]
- 15.The Joint Commission website [June 20, 2014];National Patient Safety Goals: 2014 National Patient Safety Goals. www.jointcommission.org/standards_information/npsgs.aspx.
- 16.Anthony SG, Prevedello LM, Damiano MM, et al. Impact of a 4-year quality improvement initiative to improve communication of critical imaging test results. Radiology. 2011;259:802–807. doi: 10.1148/radiol.11101396. [DOI] [PubMed] [Google Scholar]
- 17.Sistrom CL, Langlotz CP. A framework for improving radiology reporting. J Am Coll Radiol. 2005;2:159–167. doi: 10.1016/j.jacr.2004.06.015. [DOI] [PubMed] [Google Scholar]
- 18.Gershanik EF, Lacson R, Khorasani R. Critical finding capture in the impression section of radiology reports. AMIA Annu Symp Proc. 2011;2011:465–469. [PMC free article] [PubMed] [Google Scholar]
- 19.Warden GI, Lacson R, Khorasani R. Leveraging terminologies for retrieval of radiology reports with critical imaging findings. AMIA Annu Symp Proc. 2011;2011:1481–1488. [PMC free article] [PubMed] [Google Scholar]
- 20.Boland GW. Stakeholder expectations for radiologists: obstacles or opportunities? J Am Coll Radiol. 2006;3:156–163. doi: 10.1016/j.jacr.2005.10.008. [DOI] [PubMed] [Google Scholar]
- 21.Thrall JH. Reinventing radiology in the digital age. Part II. New directions and new stakeholder value. Radiology. 2005;237:15–18. doi: 10.1148/radiol.2371050258. [DOI] [PubMed] [Google Scholar]
- 22.Lacson R, Andriole KP, Prevedello LM, Khorasani R. Information from searching content with an ontology-utilizing toolkit (iSCOUT). J Digit Imaging. 2012;25:512–519. doi: 10.1007/s10278-012-9463-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform. 2001;34:301–310. doi: 10.1006/jbin.2001.1029. [DOI] [PubMed] [Google Scholar]
- 24.American College of Radiology website ACR Appropriateness Criteria. Indeterminate renal masses. [June 20, 2014];Date of origin. 1996 Last review date 2014. www.acr.org/~/media/ACR/Documents/ AppCriteria/Diagnostic/IndeterminateRenalMasses.pdf.
- 25.Bosniak MA. Should we biopsy complex cystic renal masses (Bosniak category III)? AJR. 2003;181:1425–1426. doi: 10.2214/ajr.181.5.1811425. [DOI] [PubMed] [Google Scholar]
- 26.Bosniak MA. The current radiological approach to renal cysts. Radiology. 1986;158:1–10. doi: 10.1148/radiology.158.1.3510019. [DOI] [PubMed] [Google Scholar]
- 27.Dickinson M, Ruckle H, Beaghler M, Hadley HR. Renal angiomyolipoma: optimal treatment based on size and symptoms. Clin Nephrol. 1998;49:281–286. [PubMed] [Google Scholar]
- 28.Yamakado K, Tanaka N, Nakagawa T, Kobayashi S, Yanagawa M, Takeda K. Renal angiomyolipoma: relationships between tumor size, aneurysm formation, and rupture. Radiology. 2002;225:78–82. doi: 10.1148/radiol.2251011477. [DOI] [PubMed] [Google Scholar]
- 29.Brigham and Women's Hospital website [June 20, 2014];CEBI (Center for Evidence-Based Imaging). CCTR downloads. www.brighamandwomens.org/Research/labs/cebi/CCTR/downloads.aspx. Modified June 4, 2014.
- 30.Khorasani R. Optimizing communication of critical test results. J Am Coll Radiol. 2009;6:721–723. doi: 10.1016/j.jacr.2009.07.011. [DOI] [PubMed] [Google Scholar]