O-RADS: Good Enough for Everyday Practice or a Work in Progress?

See article by Guo et al in this issue.

Deborah A. Baumgarten, MD, MPH, is a professor of abdominal radiology at the Mayo Clinic Florida. Her clinical interests include cost-effective imaging, practical and evidence-driven imaging guidelines, reducing unnecessary imaging, the genitourinary system, the thyroid, and the postoperative neck. She is a fellow of the American College of Radiology and the Society of Abdominal Radiology and volunteers with numerous national organizations, including serving on the executive committee of the American Roentgen Ray Society and the editorial board of Radiology.

In the United States, 19 880 women are expected to be diagnosed with ovarian cancer in 2022. Although rates have declined—about 3.3% each year during 2010–2019—about 1.1% of women will be diagnosed with ovarian cancer during their lifetime (1). About 10% of women will undergo surgery for an adnexal mass in their lifetime—nearly 10 times the rate of cancer (2). Therefore, it is extremely important to be able to differentiate suspicious ovarian and adnexal masses from those that can be safely ignored or followed, keeping in mind that surgery may be appropriate for some benign lesions (to remove ones that are symptomatic or to prevent future malignancy). To that end, the American College of Radiology (ACR) supported the development and dissemination of the Ovarian-Adnexal Reporting and Data System for US (O-RADS US) and for MRI (O-RADS MRI). By standardizing the lexicon for describing features of ovarian and adnexal masses and assigning risk categories for the likelihood of malignancy on the basis of an assessment of these features, management recommendations are also standardized.

In this issue of Radiology: Imaging Cancer, Guo and colleagues (3) assess interreader agreement for both O-RADS US and O-RADS MRI, as well as the intermodality agreement for O-RADS US and O-RADS MRI, in a retrospective series of 58 adnexal lesions measuring 5 cm or greater in 54 women who had undergone both US and MRI. While previous authors have reported interreader agreement for O-RADS US (4–6) or O-RADS MRI (7), the assessment of intermodality agreement is a unique feature of this paper. Two US experts and two MRI experts independently reviewed imaging and assigned O-RADS categories for assessment of interreader agreement; a third US or MRI expert adjudicated cases of disagreement for assessment of intermodality agreement and modality accuracy. Both US and MRI had high diagnostic accuracy in helping predict malignancy when using a cutoff score of O-RADS 4 (area under the receiver operating characteristic curve of 0.92 for US and 0.995 for MRI), and there was a strong, statistically significant, positive correlation between O-RADS US and MRI scores (τ = 0.72, P < .001). However, interreader agreement was only fair (κ = 0.31) for US and moderate (κ = 0.43) for MRI; intermodality agreement was also only moderate (κ= 0.58).

How can we reconcile this seeming paradox of both O-RADS US and MRI having high accuracy and a strong correlation between O-RADS US and MRI scores but at best moderate agreement between readers and modalities? There are a few points to keep in mind when interpreting the results of this study. First, this is a highly selective sample of lesions—complex lesions at least 5 cm in size—the majority of which were referred for MRI (US was performed first in 43 of 58 [74%] cases, with subsequent MRI performed to evaluate an indeterminate lesion at US). Most of the lesions that radiologists encounter in day-to-day practice (corpus lutea, hemorrhagic cysts, dominant follicles, and dermoids and endometriomas smaller than 5 cm) were excluded from analysis. Because this study was conducted at a tertiary referral center with US experts, the percentage of cases referred for US may be expected to be fewer than in a general radiology practice without US experts. In the O-RADS US system, referral to a US expert is an equivalent recommendation to performing an MRI for risk category 3 and 4 lesions. Second, O-RADS US classifies typical dermoids, endometriomas, and hemorrhagic cysts that are 10 cm or larger into O-RADS category 3, while typical dermoids, endometriomas, and hemorrhagic cysts smaller than 10 cm are put into the O-RADS 2 category. O-RADS MRI does not specify a size at which a typical dermoid, endometrioma, or hemorrhagic cyst must be upgraded to a higher O-RADS category. In this series, two lesions (one endometrioma and one cyst) were forced into the O-RADS 3 category for US but correctly downgraded to the O-RADS 2 category by using MRI; without the 10-cm size cutoff for these two lesions, US and MRI would have agreed. Third, one “governing concept” of O-RADS US is that recommendations are generally based on transvaginal sonography, although in selected cases, recommendations can be improved by transabdominal or transrectal US (8). In this series of 58 patients, eight patients did not undergo adequate transvaginal imaging (six could not tolerate it and only underwent transabdominal imaging, one was incomplete due to discomfort, and one was incomplete due to patient size). It is not clear if these are the examinations with poorer agreement. However, the figures accompanying the study demonstrate two cases of disagreement between O-RADS US and O-RADS MRI in which US overclassifies the lesions and only transabdominal US was available; thus, it is impossible to determine if the addition of transvaginal imaging would have improved the O-RADS US score. Finally, although all readers were considered experts, none were specifically trained in the use of O-RADS. Despite their availability, the ACR O-RADS US app for Android or Apple phones or an online O-RADS MRI calculator were also not used. Numerous prior validation studies, several of which used residents or other nonexperts as readers but also included a training phase, had at worst good interreader agreement, with many showing very good or near-perfect agreement (9). Therefore, it is expected that more familiarity with O-RADS, as well as the use of the ACR O-RADS US app or online O-RADS MRI risk calculator, would improve interreader agreement, in part because the risk calculators are easier to use than the tables explaining the O-RADS categories.

However, is it necessary to use O-RADS to accurately assess a lesion's likelihood of cancer? Given that an O-RADS cutoff score of 4 for US or MRI in this study had the best prediction of malignancy, does a simplified system—at least for US—make more sense? Gupta et al (10) published a paper assessing whether a simplified US approach to adnexal lesions may be sufficient. In their approach, lesions are classified into classic (simple cyst, hemorrhagic cyst, endometrioma, or dermoid) or nonclassic (anything else, including multilocular cysts, mixed cystic and solid lesions, and predominantly solid lesions) categories, and recommendations for what to do next are also simplified (reassurance for classic lesions with extremely low risk of malignancy, MRI or follow-up US for nonclassic lesions without blood flow, and referral to a gynecologic oncologist for nonclassic lesions with blood flow). Applying the simplified US system to this series of patients would result in 31 lesions being classified as O-RADS US 2 (typical dermoid, hemorrhagic cyst, endometrioma, peritoneal inclusion cyst, or hydrosalpinx); additionally, the two lesions larger than 10 cm that were forced to be categorized as O-RADS 3 (dermoid and cyst) would be considered classic and result in no further workup. This would leave 25 lesions that would be potentially malignant—of which eight were in the study. Using the simplified system would result in no malignancy being missed in this series and far fewer referrals to MRI (25 lesions at most vs the 43 in which MRI followed US) or a gynecologic oncologist.

Overall, this information suggests that if a radiologist chooses, or if it is mandated by their practice, to rely on published guidelines when assessing ovarian and adnexal lesions and making further recommendations, then education in the use of O-RADS is paramount, especially in differentiating O-RADS 1–3 lesions from O-RADS 4–5 lesions, given that the best O-RADS accuracy for predicting malignancy uses a cutoff score of O-RADS 4. Using the ACR O-RADS US app or online O-RADS MRI risk calculator is also recommended. Furthermore, to continue to hone one's skills and improve confidence in using O-RADS, taking the time to follow up on any adnexal lesion for which an O-RADS score was assigned is important given the variability in assigning scores, even among US and MRI experts. If a radiologist finds O-RADS US too onerous, relying on the simplified system proposed by Gupta et al (10) also seems perfectly reasonable.