Clinician Nudge to Gynecologic Oncology Referral at Suspected Ovarian Cancer Diagnosis: A Pilot Study

Abstract

Introduction

Only two-thirds of patients with ovarian cancer ever see a gynecologic oncologist. Our objective was to examine the feasibility of an electronic health record-based nudge to clinicians for referral to gynecologic oncology at suspected ovarian cancer by imaging.

Methods

We developed a nudge, a short behavioral economics informed best practice advisory with a pended referral order for gynecologic oncology, for primary care, emergency medicine, and obstetrician/gynecology clinicians for when a patient had a O-RADS 4 or 5 lesion on imaging and had not already seen gynecologic oncology. In 2024, clinicians were sent the nudge within 2 business days of a patient’s abnormal imaging through the electronic health record. Our primary outcome was referral rate to gynecologic oncology compared to a historic cohort of patients with O-RADS 4 or 5 lesions from 2020-2023.

Results

In this prospective cohort study, we sent 20 clinician nudges for gynecologic oncology referral; six clinicians (30%) responded that the nudge changed their referral behavior. The 90-day referral rate was 75% compared to historic baseline of 61%. In the pilot, 92% patients undergoing surgery for complex adnexal mases had surgery with gynecologic oncology compared to historic baseline of 82%. One in four patients in the pilot were diagnosed with cancer, all early-stage disease.

Conclusions

A clinician nudge for gynecologic oncology referral at suspected ovarian cancer diagnosis was acceptable and associated with 75% referral rate. A clinician nudge standardizes gynecologic oncology referral and may improve early detection of ovarian cancer. A randomized controlled trial of the clinician nudge is warranted.

Plain Language Summary

Patients with ovarian cancer live longer when treated by a gynecologic oncologist, a surgeon whose focus is gynecologic tumors. Yet one-third of patients with ovarian cancer are never seen by a gynecologic oncologist. In this pilot study, we identified clinicians whose patients had imaging concerning for ovarian cancer. We sent these clinicians a message recommending referral to gynecologic oncology with a pended referral order, also known as a clinician nudge. We found that a clinician nudge to gynecologic oncology referral increased referral rate of suspected ovarian cancer to 75%. Clinicians indicated that the nudge was acceptable, and 30% changed their referral behavior after the nudge. More patients had surgery with gynecologic oncology after the nudge than similar patients not in the study. One in four patients in the pilot were diagnosed with cancer, all early-stage disease. A clinician nudge makes it easy and straightforward to clinicians to refer patients to gynecologic oncology. A nudge may improve early detection of ovarian cancer, and a larger study is warranted.

Key Messages

What is already known on this topic

Referral to gynecologic oncology is associated with improved survival in ovarian cancer, but one-third of patients never seen a gynecologic oncologist.

What this study adds

A best practice advisory nudge to gynecologic oncology referral increased referral rate of complex adnexal masses to 75%. Clinicians indicated that the nudge was acceptable and 30% changed their referral behavior after the nudge.

How this study might affect research, practice or policy

A clinician nudge standardizes gynecologic oncology referral and may improve early detection of ovarian cancer.

Introduction

In ovarian cancer, patients seen at least once by a gynecologic oncologist live an average of 12 months longer than patients treated solely by other specialties, due to higher receipt of optimal chemotherapy and surgery.^1-4 Yet one-third of patients never see a gynecologic oncologist. ⁵ Referral rates^6-9 are even lower for the 28% of patients with ovarian cancer who identify as Black, Hispanic, Asian, or other races and for the 23% who have low incomes, shortening survival. When referred to gynecologic oncology, patients from historically marginalized communities have 2-3 times longer wait times, contributing to later stage at diagnosis. ¹⁰

Given the lack of effective screening for ovarian cancer, improving referral at radiologic or pathologic diagnosis is the most realistic opportunity to improve outcomes and reduce disparities that compound throughout cancer care. Since the early 2000s, all leading organizations—the Centers for Disease Control and Prevention (CDC), ¹¹ the Society for Gynecologic Oncologists, the American College of Obstetricians and Gynecologists, ¹² the American Society of Clinical Oncology, ¹³ and the National Comprehensive Cancer Network ¹⁴ —have consistently recommended that patients with imaging suspicious for ovarian cancer have a surgical evaluation with a gynecologic oncologist prior to surgery or starting chemotherapy and that any surgery, including laparoscopic biopsies and oophorectomy, should be performed by a gynecologic oncologist. Despite the known benefits of specialist involvement and documented room for improvement in equitable implementation of evidence-based care, a systematic review by CDC identified no studies focused on increasing referral to gynecologic oncology in the US. ¹⁵

Suboptimal referral rates are particularly concerning because there are clear criteria for who should be referred. In 2020, the American College of Radiology launched the Ovarian-Adnexal Imaging-Reporting-Data System (O-RADS).^16,17 Like the BI-RADS system in breast cancer, O-RADS uses standardized criteria to classify the risk of ovarian cancer in adnexal masses, with 95% sensitivity and specificity.^18,19 The 5-point scale recommends gynecologic oncology referral with an O-RADs score of 4 or 5. Prior to O-RADS, there were no standardized criteria for radiologic diagnosis of ovarian cancer, leading to lack of clarity in need for referral. Introduction of a standardized imaging system similar to O-RADS, and a streamlined referral system, in the United Kingdom improved gynecologic oncology referral from 11% to 86% and reduced patient wait time and anxiety.^20,21

Nudges, a behavioral economics-informed implementation strategy, adjust how choices are presented within the electronic health record (EHR) and facilitate (“nudge”) the desired behavior by introducing or reinforcing a treatment standard (ie, referral) and providing an automatic option.^22-26 Nudges offer salient prompts at relevant points in the workflow and are designed to overcome such biases. They can be especially potent when they refine the “choice architecture,” or how a choice is presented. “Changing the default option” (ie, low or inconsistent tendency to refer), and minimizing the effort required to make the evidence-based choice can streamline decision-making in a way that can also reduce implicit bias and disparities for historically marginalized groups.^23,24 Active choice nudges have been shown to improve referral for breast, colorectal, and cervical cancer screening and evidence-based treatment decision-making in other areas of oncology.^23,27-29

Our objective was to see if a clinician nudge at suspected ovarian cancer diagnosis (ie, O-RADS 4 or 5 lesion on imaging) is feasible and could improve referral to gynecologic oncology in a pilot study.

Methods

In this prospective cohort study with study arm intervention, we piloted a clinician nudge to gynecologic oncology referral and followed patients for 180 days after abnormal imaging. The study was reviewed and approved by the University of Pennsylvania Institutional Review Board on December 13, 2024 (#854677). Given there was no direct patient contact, the study operated under a waiver of informed consent. It was registered on clinicaltrials.gov (#NCT06451263) on May 31, 2024. The study was conducted in accordance with the Helsinki Declaration of 1975, as revised in 2024.

Study population

We defined eligible patients as those with imaging findings suspicious for ovarian cancer not previously seen in gynecologic oncology. We defined imaging suspicious for ovarian cancer as radiologist reported Ovarian-Adnexal Reporting and Data System (O-RADS) score of 4 (10-49% risk of ovarian cancer) or 5 (≥50% risk) on ultrasound or magnetic resonance imaging (MRI).^16,17 O-RADS scoring was implemented universally with the healthy system in 2022 with a standardized template; the template had been used at the largest imaging site since 2020. The American College of Radiology recommends gynecologic oncology referral for O-RADS 4 or 5 lesions; after referral, lesions can be managed solely by a gynecologic oncologist or in collaboration with a benign gynecologist. We used natural language processing of radiology reports to identify consecutive patients with O-RADS 4 or 5 lesions. Our natural language processing recognized the template as well as alternative spellings of O-RADS (eg, orads, ORAD).

We excluded patients meeting the following criteria: (1) the patient had been seen in gynecologic oncology in the past 5 years, (2) the imaging was ordered by a University of Pennsylvania Health System (UPHS) gynecologic oncologist, and (3) the ordering clinician was outside the UPHS system (ie, not available through EHR messaging). Patients referred to gynecologic oncology within 24 hours of abnormal imaging or admitted to a UPHS hospital at time of abnormal imaging (ie, inpatient consults) were also excluded. Nudges were sent until 20 consecutive patients had been included.

Intervention

The active choice nudge was designed and refined with feedback from UPHS primary care clinicians as well as the Penn Medicine Nudge Unit (Figure 1). It used multiple principles of behavioral economics to activate clinician behavior, including tailoring (ie, “Your patient, [Name]”), leadership endorsement (ie, Penn Medicine and national guidelines recommend referral to gynecologic oncology”), and requesting justification for deviation from evidence-based practice (ie, If you would not like to make the referral, please message back your justification for not referring to gynecologic oncology: Patient refused, patient already in gynecologic oncology care, not clinically appropriate, or other”). The nudge was sent via EHR message to the clinician who ordered the abnormal imaging. A pended order for referral to gynecologic oncology was included in the nudge message to minimize clinician effort (ie, default option with active choice or opt-in).

Figure 1.

Clinician nudge for gynecologic oncology referral

During the study period, all new imaging with an O-RADS score pulled into a password-protected electronic database daily. The study coordinator screened the database daily to identify patients with O-RADS score of 4 or 5. For eligible patients, we sent the clinician who ordered the abnormal imaging a standardized nudge with default pended order for referral to gynecologic oncology through the EHR within 2 business days of abnormal imaging.

If the referral order was signed, the gynecologic oncology scheduling team contacted patients within 2 business days of referral placement. Over the phone, they would schedule the patient for a visit at one of 5 gynecologic oncology clinics.

Comparator Cohort

We compared outcomes from the pilot to the cohort of patients with O-RADS 4 or 5 lesions in the health system. To parallel the pilot exclusion criteria, we excluded patients not established in gynecologic oncology from 2020-2023 (n = 47), patients with <24 hours from imaging to referral (38), <24 hours from imaging to visit (1), and inpatient consults (12). There were no major changes in care or referral pathways during the time period that would threaten comparability.

Outcomes

The primary outcome was referral rate to gynecologic oncology compared to a historical cohort of patients with O-RADS 4/5 lesions on outpatient imaging in 2020-2023 (n = 331). Secondary outcomes include completion of gynecologic oncology visit and care delivery, including surgery, pathologic diagnosis, and stage of cancer at diagnosis. We followed patients in the nudge pilot for 180 days after abnormal imaging. We defined time to care delivery starting with the date imaging was performed (eg,. Time to surgery is date of imaging performed to date of surgery). For acceptability, we collected clinician read receipt and any free-text reply to EHR nudge.

Statistical Analysis

We report descriptive characteristics of nudge receipt and clinician responses to the nudge. We describe nudge cohort compared to the historical cohort using chi-squared tests. In accordance with the journal’s guidelines, we will provide our data for independent analysis by a selected team by the Editorial Team for the purposes of additional data analysis or for the reproducibility of this study in other centers if such is requested. The reporting of this study conforms to the STROBE guidelines. ³⁰ All patient details were de-identified.

Results

Over 3 months in 2024, we identified 60 patients with O-RADS 4 or 5 lesion on imaging. Twenty-two had been already been seen in gynecologic oncology, leaving 38 potentially eligible for intervention. Of these, 4 patients were referred to gynecologic oncology within 24 hours of abnormal imaging prior to nudge screening, and 5 had ordering providers not available through the EHR. Nine patients had a 14-day delay in O-RADS score importing into our database due to technical issues with radiology reports from one site importing into our database, leaving 20 eligible patients.

For the 20 eligible patients, we sent 20 clinician nudges for gynecologic oncology referral. The mean and median times from imaging to nudge delivery was 3 days (range 1-10 days). Nineteen of 20 nudges (95%) were read by the ordering clinician. Following nudges, 15 patients were referred to gynecologic oncology within 90 days of abnormal imaging, a 75% referral rate in the pilot compared to historic baseline of 61% referral. Of the 5 patients not referred, 3 clinicians sent back a justification for no referral as “not clinically appropriate,” and 2 did not respond. Six of 20 (30%) clinicians who received the nudge sent back a message indicating the nudge changed their decision-making, ie, they would not have referred the patient to gynecologic oncology without the nudge. In the nudge pilot, the median time from imaging to referral was 10 days (range 2-90 days).

Like the historic cohort, the majority of clinicians in the pilot were general obstetrician-gynecologists (11, 55%), family or internal medicine clinicians (6, 30%), or emergency medicine clinicians (3, 15%). Patients in the pilot cohort were less likely to have a primary care clinician than patients in the historic cohort (Table 1), but were otherwise similar. The median age was 55 years (interquartile range 42-65) in the pilot and 49 (IQR 36-62) in the historic cohort. In both, the majority of abnormal imaging was from ultrasound (90% pilot vs 93% historic) and scored as O-RADS 4 (90% vs 85%).

Table 1.

Comparison of Patients in Nudge Pilot and Historic Cohort

Demographics	Nudge pilot (n = 20)	Historic cohort (n = 331)	p-value
Age in years, median (interquartile range)	55 (42-65)	49 (36-62)	0.16
Postmenopausal (age ≥55 years)	10 (50%)	121 (37%)	0.24
Race
White	15 (75%)	222 (67%)	0.89
Black	3 (15%)	68 (21%)
Asian	1 (5%)	14 (4%)
Some other race	1 (5%)	25 (8%)
Hispanic ethnicity	2 (10%)	22 (7%)	0.57
Preferred language other than English	0	17 (5%)	0.30
Insurance
Uninsured	0	7 (2%)	0.87
Medicaid	4 (20%)	42 (13%)
Medicaid-medicare	0	4 (1%)
Medicare (traditional and advantage)	5 (25%)	71 (22%)
Private	11 (55%)	202 (61%)
VA or tricare	0	3 (1%)
Rural residence	0	19 (6%)	0.27
High-poverty residence (≥20% of population in zip code below federal poverty line)	3 (15%)	53 (16%)	0.90
Historically marginalized	9 (45%)	154 (47%)	0.88
Imaging modality
US	18 (90%)	306 (93%)	0.12
MRI	1 (5%)	21 (6%)
CT	1 (5%)	2 (1%)
O-RADS score
4	18 (90%)	278 (85%)	0.51
5	2 (10%)	51 (16%)
Ordering specialty
Ob/gyn	11 (55%)	229 (70%)	0.32
Internal medicine	1 (5%)	31 (9%)
Family medicine	5 (25%)	39 (12%)
Emergency medicine	3 (15%)	30 (9%)
Patient has primary care clinician	13 (65%)	279 (85%)	0.02

Historic cohort includes patients with O-RADS (ovarian-adnexal reporting and data system) 4 or 5 lesions in the health system not established in gynecologic oncology from 2020-2023 (n = 373), excluding patients with <24 hours from imaging to referral (38), <24 hours from imaging to visit (1), and inpatient consults (12).Historically marginalized was defined as living in rural or high-poverty residence, being uninsured, having medicaid insurance, having a preferred language other than English, or self-identifying as hispanic, black, asian, or other racial group.P-value is from chi-squared comparison of patients in pilot and historic cohort. When numbers were <5, we collapsed categories for comparison (eg, ob/gyn vs other ordering specialties, private vs other insurance types).

The proportion of patients with a gynecologic oncology visit within 120 days of abnormal imaging was 60% (80% among those referred) similar to the historic baseline of 58%. There was a median of 29 days (range 12-106) from abnormal imaging to first gynecologic oncology visit. In the pilot, one patient referred to gynecologic oncology declined a visit when called by the gynecologic oncology scheduling office; the two others referred were contacted with no visit scheduled. Both had benign appearing masses on follow-up MRI imaging and may have declined the referral.

In the nudge pilot, eleven patients had surgery with gynecologic oncology, four of whom had cancer diagnosed at surgery. In the historic cohort, 82% of patients had surgery with a gynecologic oncologist compared to 92% in the pilot, driven by the higher referral rate. The median time from first gynecologic oncology visit to surgery was 19 days (range 9-68) in the pilot compared to 27 days (4-730 days) in the historic cohort.

The incidence of cancer diagnosed at surgery was similar in the pilot (4; 33%) and historic cohorts (53; 28%). All the cancers in the pilot diagnosed were Stage 1; three had Stage 1A ovarian cancer, and one had Stage 1 appendiceal cancer. One patient not referred to gynecologic oncology had adnexal surgery with a benign obstetrician-gynecologist: pathology was benign. In the historic cohort, of the 36 patients diagnosed with ovarian cancer, 83% (30) of patients had Stage 1 ovarian cancer at diagnosis. Of the 17 patients diagnosed with non-ovarian cancer at surgery in the historic cohort, 14 had uterine cancer, 1 had cervical cancer, and 2 had colon cancer: 11 had stage 1 cancer of their respective primary (65%) for a total of 77% (41) Stage 1 cancers among those diagnosed in the historic cohort.

Discussion

Main Findings

A clinician nudge for gynecologic oncology referral at suspected ovarian cancer by imaging was acceptable and associated with 75% referral rate. In a small pilot study of the nudge, one in four patients were diagnosed with cancer, all early-stage disease. A clinician nudge standardizes gynecologic oncology referral and may improve early detection of ovarian cancer; further evaluation in a larger trial is warranted.

Comparison with Existing Literature

The magnitude of our pilot findings is consistent with those of active choice or opt-in nudges in other areas of medicine.^29,31 Opt-out nudges in which referral order proceeds unless clinician actively declines the order typically have a stronger behavioral effect (ie, increased referral rate).^24,28 We chose an “active choice” (ie, pended referral) to preserve clinician autonomy and maintain patient-clinician relationships so that clinicians could communicate results in their preferred manor and timeline with patients. Given the wide range of time to referral seen in our pilot, follow-up nudges or a timed opt-out (ie, automated referral 60 days after abnormal imaging) may be necessary to decrease time to referral and further improve early detection and treatment of ovarian cancer. In addition, as 3 patients in the pilot appear to have declined the referral to gynecologic oncology, a patient component to the nudge may be beneficial to increase referral uptake. A patient-directed nudge could include education about ovarian cancer risk and the role of gynecologic oncology in work-up of complex adnexal masses.

Thirty percent of clinicians receiving the nudge indicated it changed their behavior, suggesting the nudge addressed a gap in clinician knowledge. A qualitative study with primary care clinicians and medical oncologists (outside of Penn Medicine) found variability in clinician risk assessment and scheduling as main barriers to gynecologic oncology referral. ³² Referral rates from primary care to gynecologic oncology vary 6-fold between practices, with older and publicly-insured patients less likely to be referred.^10,33,34 Since most primary care clinicians and obstetrician/gynecologists will see few cases of ovarian cancer in their careers, clinician knowledge gaps, compounded by implicit bias (eg, the perception that a patient will be unlikely to complete care due to patient or system barriers), are a key driver of no referral and delayed referral. In a US cohort of patients with ovarian cancer, only 39% were referred directly to gynecologic oncology after suspicious imaging and 49% saw 1 or more specialists (eg, a gastroenterologist, general surgeon) prior to a gynecologic oncologist. ³⁴ Two-thirds of mis-referred patients experienced treatment delays.^34-36 In our pilot, while we did not see a major change in time to treatment, we saw increased proportion of patients with complex adnexal masses undergoing surgery with a gynecologic oncologist, which has been associated with improved staging and survival in prior studies.

Strengths and Limitations

This was a pilot study testing acceptability and feasibility of a clinician nudge for gynecologic oncology referral in a tertiary health system with an integrated electronic health record, including radiology reports and electronic referral orders. Clinical outcomes reported are exploratory and need validation in larger study. Patients in the pilot were less likely to have a primary care clinician than those in the historic cohort, which may have biased pilot findings toward the null. Having a primary care clinician may be associated with improved cancer outcomes through enhanced follow-up of abnormal imaging, such as O-RADS.

With 11 gynecologic oncologists and 5 gynecologic oncology clinics, we had staffing to absorb increased clinical volume from the nudge pilot. We built a dashboard outside the EHR to identify patients within 24 hours of O-RADS 4 or 5 results, which required programming expertise and ongoing refinement as evidenced by 9 patients experiencing 14-day in dashboard entry (and thus excluded from study). The feasibility of a clinician nudge may differ in health systems with more limited gynecologic oncology and/or programming capacity. In addition, while O-RADS adoption is increasing, its use is not universal and currently limited to ultrasound and MRI. Development of O-RADS scoring system for CT, an imaging modality often used for evaluation of pelvic or abdominal symptoms, would facilitate appropriate triage of additional masses suspicious for ovarian cancer. However, once patients were identified, the nudge delivery was simple with EHR message and pended order. The mode of delivery of a nudge could be adapted to a non-EHR platform, such as email to clinician and faxed referral, or to a non-O-RADS based scoring system in other health systems.

Conclusion

A clinician nudge standardizes gynecologic oncology referral and may improve early detection of ovarian cancer. A randomized controlled trial of the clinician nudge for gynecologic oncology referral is warranted.

Appendix.

Abbreviations

CDC:

Centers for disease control and prevention

EHR:

Electronic health record

MRI:

Magnetic resonance imaging

O-RADS:

Ovarian-adnexal reporting and data system

UPHS:

University of pennsylvania health system

ORCID iD

Anna Jo Bodurtha Smith https://orcid.org/0000-0002-7692-773X

Ethics Considerations

The study was reviewed and approved by the University of Pennsylvania Institutional Review Board on December 13, 2024 (#854677) and registered on clinicaltrials.gov (#NCT06451263).

Consent to Participate

Given there was no direct patient contact, the study operated under a waiver of informed consent.

Consent for Publication

All authors reviewed and approved the manuscript for publication.

Data Availability Statement

The authors will provide the original data on written request.*