Addressing algorithmic bias in lung cancer screening eligibility

Abstract

Background

The US Preventive Services Task Force (USPSTF) lung cancer screening eligibility guidelines and proposed risk models have been developed using data predominantly from White populations. Studies show that these eligibility strategies perform inconsistently across racially diverse populations, suggesting evidence of algorithmic bias. We assessed several lung cancer screening eligibility strategies and explored how algorithmic bias can be resolved to improve equity in eligibility.

Methods

Using the Southern Community Cohort Study, a large US study of predominantly Black/African American individuals, we evaluated the performance of 8 existing lung cancer screening eligibility strategies (USPSTF 2021; American Cancer Society 2023 recommendations; USPSTF_{SmokeDuration}; Prostate, Lung, Colorectal and Ovarian 2012 risk prediction model [PLCOm2012]; PLCOm2012_NoRace; PLCOm2012_Update; Lung Cancer Risk Assessment Tool; and Lung Cancer Death Risk Assessment tool) and 2 new race-aware strategies proposed by our team (USPSTF_RaceSpecific and PLCOm2012_RaceSpecific).

Results

Among 52 667 adults (65% Black/African American, 31% White, 4% Multiracial/Other) with a smoking history, 1689 developed lung cancer over 15 years. Most screening strategies identified fewer Black/African American participants who developed lung cancer as eligible for screening vs their White counterparts (sensitivity for Black/African American individuals = 0.46-0.73 vs 0.72-0.80 for their White counterparts). Racial eligibility disparities were not resolved by removing race, removing the “years since quit” criterion, or using uniform risk thresholds. Replacing pack-years with smoking duration improved equity but overinflated the false-positive rate (0.71 for Black/African American persons vs 0.61 for White persons). Instead, race-aware approaches that tailored eligibility thresholds by race yielded the best sensitivity-specificity trade-off and minimized inequities (sensitivity = 0.71-0.73 for Black/African American persons vs 0.72-0.74 for White persons; false-positive rate = 0.49-0.50 for Black/African American persons vs 0.50-0.53 for White persons).

Conclusion

Our findings suggest that race-aware approaches are necessary to address algorithmic bias and ensure equitable opportunities for lung cancer screening.

Introduction

With growing concern about fairness in medicine, clinical algorithms and risk prediction models are decision-support tools that can help ensure that all patients are assessed equally. Recent emphasis on reexamining the role of race in clinical algorithms has shown that careful thought and justification in algorithm selection is needed to reduce racial disparities.^1-4 Algorithmic bias occurs when a model inherits properties of the training data (eg, skewed data or unrepresentative sampling) that do not generalize well to the target population.^5,6 This bias can reinforce the very disparities we aim to eliminate. Here, we show that the US Preventive Services Task Force (USPSTF) lung cancer screening eligibility guidelines and other models proposed for determining screening eligibility exhibit racial disparities indicative of algorithmic bias. One way to address this is bias to fine-tune the algorithm to minimize the disparity in diagnostic performance. We propose a fine-tuning approach that ultimately results in diagnostic performance that minimizes observed disparities in lung screening eligibility.

In the United States, eligibility for lung cancer screening is determined by USPSTF guidelines based on age and smoking history.⁷ There is growing recognition, however, that these guidelines may not optimally identify for screening individuals at highest risk. It is well documented that Black/African American individuals have among the highest lung cancer incidence and mortality rates in the United States and tend to be diagnosed with lung cancer at an earlier age than White individuals.^8-12 Even after accounting for cumulative smoking exposure, Black/African American individuals can be as much as 61% more likely than their White counterparts to develop lung cancer, particularly at low levels of smoking intensity.^8,13 Yet, up to 67% of Black/African American individuals who develop lung cancer are ineligible for screening based on age and smoking history criteria.^14,15 Although race is not the cause of increased risk, it remains a helpful predictor of observed differences in lung cancer incidence.

Several modifications to improve screening eligibility have been proposed.^16-18 Yet these modifications, much like the USPSTF guidelines, have been developed using data collected predominantly from White populations^19-21 and tend not to reflect known population differences in smoking behaviors and lung cancer risk.^8,9,13,22-25 For example, Black/African American individuals often report low intensities and long durations of smoking (ie, they consume fewer cigarettes per day but smoke for longer durations than their White counterparts).^26,27 They are also more likely to smoke currently and have fewer attempts to quit smoking.²⁸ In addition, the median smoking pack-years among Black/African American individuals who develop lung cancer is substantially lower than that of their White counterparts.^14,15 Guidelines and models trained on data from predominantly White populations do not appropriately account for these important differences in smoking patterns and lung cancer incidence, resulting in inconsistent and inefficient performance across different race groups—evidence of algorithmic bias. For example, Pu et al.²⁹ concluded that the USPSTF 2021 guidelines and the Prostate, Lung, Colorectal and Ovarian 2012 risk prediction model²¹ (PLCOm2012) could eliminate screening eligibility disparities between Black/African American individuals and White individuals. Subsequent examination of these models in the more racially diverse Multiethnic Cohort study, however, yielded a different conclusion.¹⁴ Given this and other contradictory findings,^30-32 further work is needed to assess variation in lung cancer screening eligibility across populations.

In this article, we sought to evaluate the impact of algorithmic bias on several proposed lung cancer screening eligibility strategies using the Southern Community Cohort Study (SCCS), the largest prospective US cohort of predominantly Black/African American smoking adults. We assessed eligibility across race and sex using 3 risk factor–based strategies, 5 risk model–based strategies, and a novel approach for fine-tuning these strategies to yield equal diagnostic performance across race groups. Using the USPSTF Health Equity Framework^33,34 and its guidance on bias and fairness in clinical prediction algorithms,^5,35 we discuss the trade-off in performance characteristics between the strategies.

Methods

Study population

The SCCS is a prospective observational cohort study investigating health disparities with participants mainly recruited from community health centers across 12 southeastern US states.^36,37 Study design details are available elsewhere.³⁶ Briefly, approximately 85 000 adults aged 40-79 years who did not receive cancer treatment (except for nonmelanoma skin cancer) within the previous 12 months were enrolled between March 2002 and September 2009 and followed up until December 2016 for various health outcomes. The SCCS was approved by institutional review boards at Vanderbilt University (Nashville, TN) and Meharry Medical College (Nashville, TN) (IRB No. 010345). Written informed consent was obtained from all participants.

Incident lung cancer, smoking behavior, and additional risk factor ascertainment

Incident lung cancer cases (International Classification of Diseases for Oncology codes C340-349) were identified through December 31, 2016, through links with the National Death Index and cancer registries in the 12-state SCCS catchment area. Demographics and smoking behaviors were collected in person or through mailed-in questionnaires, including age, sex, self-identified race and ethnicity, educational attainment, body mass index, household income, personal medical history, and family history of lung cancer.

We used information from the most recent follow-up survey or, for participants with lung cancer, the follow-up survey nearest to but before the time of diagnosis. Smoking intensity was measured in smoking pack-years. For current smokers who were missing follow-up smoking information, we carried forward the last recorded smoking status and assumed that the number of cigarettes smoked per day remained the same and pack-years increased, as implemented in our prior work.¹⁵ Each participant’s follow-up period was divided into a maximum of 3 time windows based on the dates of the follow-up surveys and when the lung cancer diagnosis or most recent follow-up occurred. Pack-years were estimated within each time window. If a participant started or stopped smoking within that time window, midpoint ages were used to estimate the pack-years. Quit duration was calculated as years from smoking cessation until lung cancer diagnosis or most recent follow-up.

Statistical analysis

Only adults with a history of smoking were included in our analyses. We grouped screening strategies into (1) risk factor–based methods (USPSTF 2021 guidelines,⁷ USPSTF_{SmokeDuration}¹⁶ [which uses 20 years smoked instead of 20 pack-years], and American Cancer Society [ACS] 2023 recommendations¹⁷ [which removes the “years since quit” criterion]) and (2) risk model–based methods (PLCOm2012²¹ [a 6-year lung cancer incidence model], PLCOm2012_NoRace³⁰ [removes race as a predictor], PLCOm2012_Update³¹ [recalibrates the race coefficient], the Lung Cancer Risk Assessment Tool [LCRAT,³⁸ a 5-year lung cancer incidence model], and the Lung Cancer Death Risk Assessment Tool [LCDRAT,³⁸ a lung cancer mortality model]). For all strategies, the eligibility of SCCS participants for lung cancer screening was assessed. Details regarding risk factors and risk thresholds for eligibility determination are provided in Table S1. Subgroups were defined by self-reported race, sex, and the intersection of race and sex.

Sensitivity, specificity, and exact CIs were computed for each screening strategy. Sensitivity was defined as the proportion of participants eligible for screening among the individuals who developed lung cancer, P(eligible for screening | lung cancer). Specificity was defined as the proportion of participants ineligible for screening among those who did not develop lung cancer, P(ineligible for screening | no lung cancer). Wherever possible, receiver operating characteristic (ROC) curves were generated. Differences in sensitivity and specificity between groups were assessed using difference-in-proportion tests, with estimates and 95% CIs reported.

We compared 3 approaches for mitigating algorithmic bias: the use of (1) uniform screening thresholds, (2) race-specific screening thresholds, and (3) a blended approach combining risk factor–based and risk model–based strategies. Using race-specific thresholds is an example of a race-aware⁵ fine-tuning approach that resolves algorithmic bias in diagnostic performance. Race-aware fine-tuning of eligibility thresholds accommodates racial disparities, which can be driven by structural factors associated with race rather than by race itself. In this setting, a systemic lack of racial diversity in training data used to develop lung cancer risk prediction models has resulted in eligibility thresholds that do not generalize to all race groups.^8,13 As a result, a uniform eligibility threshold leads to differential diagnostic performance across race groups. To address this differential, we varied screening eligibility thresholds by race to force the guidelines/risk prediction models to perform in a race-neutral manner (ie, to yield equal diagnostic performance across race groups). We implemented this approach by modifying the USPSTF 2021 guidelines to match its sensitivity among Black/African American individuals to its sensitivity among White individuals using different pack-year thresholds by race, hereafter named USPSTF_RaceSpecific. We applied a similar approach to the PLCOm2012 risk model using different eligibility risk thresholds by race, hereafter named PLCOm2012_RaceSpecific. We chose to equalize sensitivity to prioritize the identification of individuals with lung cancer for screening. This approach, however, can be applied to any diagnostic measure of interest (eg, specificity, or a weighted average of sensitivity and specificity). Analyses were completed using R, version 4.3.1, statistical software (R Foundation for Statistical Computing).

Results

Demographics and race-sex differences in smoking behaviors at baseline

Among 52 667 SCCS participants with a smoking history, 64.5% self-identified as Black/African American and 31.4% self-identified as White. The remaining approximately 4.0% were combined into a “Multiracial/Other” category; individuals self-identified as Asian or Pacific Islander, Hispanic/Latino, American Indian or Alaska Native, Mixed Race, or Other Race. About half the participants were female, approximately two-thirds reported their highest level of educational attainment as high school or less, and nearly 60% reported an annual household income of $15 000 or less (Table 1). A total of 1689 (3.2%) participants were diagnosed with lung cancer over the 15-year follow-up period. Smoking behaviors differed greatly by race and sex. Black/African American men reported the highest percentage of current smoking (72.9%) at baseline, whereas White men reported the lowest percentage (54.6%). The median pack-years smoked was lowest among Black/African American women and highest among White men (12.5 vs 31.0). Among participants who formerly smoked, White men had the longest median quit duration (20 years).

Table 1.

Baseline demographic characteristics and smoking behaviors among ever-smoking SCCS participants, by race and sex.

	Black/African American female (n = 16 467)	Black/African American male (n = 17 366)	Multiracial/Other female (n = 1221)	Multiracial/Other male (n = 898)	White female (n = 9313)	White male (n = 7158)	Total (N = 52 423)
Incident lung cancer, No. (%)
No lung cancer	16 042 (97.4)	16 773 (96.6)	1185 (97.1)	873 (97.2)	8968 (96.3)	6893 (96.3)	50 734 (96.8)
Lung cancer	425 (2.6)	593 (3.4)	36 (2.9)	25 (2.8)	345 (3.7)	265 (3.7)	1689 (3.2)
Age at enrollment, median (range), y	50.0 (40.0-79.0)	50.0 (40.0-79.0)	50.0 (40.0-78.0)	51.0 (40.0-78.0)	51.0 (40.0-79.0)	53.0 (40.0-79.0)	50.0 (40.0-79.0)
Age at Diagnosis, median (range), ya	60.0 [42.0, 86.0]	60.0 [42.0, 88.0]	64.0 [48.0, 74.0]	61.0 [48.0, 83.0]	63.0 [44.0, 85.0]	65.0 [40.0, 87.0]	61.0 [40.0, 88.0]
Highest education level attained, No. (%)
Less than high school	5513 (33.5)	6163 (35.5)	301 (24.7)	242 (26.9)	2579 (27.7)	1831 (25.6)	16 629 (31.7)
High school	5342 (32.4)	6276 (36.1)	328 (26.9)	246 (27.4)	3121 (33.5)	2279 (31.8)	17 592 (33.6)
Above high school	5603 (34.0)	4916 (28.3)	592 (48.5)	410 (45.7)	3611 (38.8)	3043 (42.5)	18 175 (34.7)
Missing	9 (0.1)	11 (0.1)	0 (0)	0 (0)	2 (0.0)	5 (0.1)	27 (0.1)
Annual household income, No. (%)
<$15 000	10 511 (63.8)	10 926 (62.9)	658 (53.9)	461 (51.3)	5134 (55.1)	3303 (46.1)	30 993 (59.1)
$15 000-$25 000	3548 (21.5)	3571 (20.6)	249 (20.4)	185 (20.6)	1792 (19.2)	1269 (17.7)	10 614 (20.2)
>$25 000	2199 (13.4)	2692 (15.5)	283 (23.2)	238 (26.5)	2264 (24.3)	2496 (34.9)	10 172 (19.4)
Missing	209 (1.3)	177 (1.0)	31 (2.5)	14 (1.6)	123 (1.3)	90 (1.3)	644 (1.2)
Chronic obstructive pulmonary disease, No. (%)
No	14 684 (89.2)	16 359 (94.2)	976 (79.9)	783 (87.2)	7408 (79.5)	6201 (86.6)	46 411 (88.5)
Yes	1752 (10.6)	988 (5.7)	243 (19.9)	111 (12.4)	1882 (20.2)	949 (13.3)	5925 (11.3)
Missing	31 (0.2)	19 (0.1)	2 (0.2)	4 (0.4)	23 (0.2)	8 (0.1)	87 (0.2)
Family history of lung cancer, No. (%)
No	14 937 (90.7)	16 228 (93.4)	1066 (87.3)	811 (90.3)	7880 (84.6)	6203 (86.7)	47 125 (89.9)
Yes	1530 (9.3)	1138 (6.6)	155 (12.7)	87 (9.7)	1433 (15.4)	955 (13.3)	5298 (10.1)
Insurance coverage, No. (%)
No insurance	6684 (40.6)	8499 (48.9)	471 (38.6)	379 (42.2)	3820 (41.0)	2763 (38.6)	22 616 (43.1)
Insurance	9669 (58.7)	8785 (50.6)	737 (60.4)	516 (57.5)	5431 (58.3)	4375 (61.1)	29 513 (56.3)
Missing	114 (0.7)	82 (0.5)	13 (1.1)	3 (0.3)	62 (0.7)	20 (0.3)	294 (0.6)
Body mass index, median (range)	30.5 [12.9, 84.5]	26.1 [15.2, 71.0]	29.3 [14.2, 67.3]	27.5 [15.8, 66.0]	29.0 [13.2, 71.7]	27.3 [15.2, 67.8]	28.0 [12.9, 84.5]
Missing, No. (%)	240 (1.5)	115 (0.7)	13 (1.1)	7 (0.8)	109 (1.2)	46 (0.6)	530 (1.0)
Smoking status, No. (%)
Current	10 226 (62.1)	12 656 (72.9)	719 (58.9)	555 (61.8)	5422 (58.2)	3910 (54.6)	33 488 (63.9)
Former	6241 (37.9)	4710 (27.1)	502 (41.1)	343 (38.2)	3891 (41.8)	3248 (45.4)	18 935 (36.1)
Cigarettes/day, median (range)	10.0 [0.500, 200]	10.0 [0.500, 130]	12.0 [0.500, 200]	20.0 [0.500, 140]	20.0 [0.500, 200]	20.0 [0.500, 200]	10.0 [0.500, 200]
Missing, No. (%)	188 (1.1)	137 (0.8)	14 (1.1)	10 (1.1)	69 (0.7)	47 (0.7)	465 (0.9)
Pack-years, median (range)	12.5 [0.0125, 250]	15.6 [0.0125, 268]	16.5 [0.0125, 460]	22.5 [0.0500, 215]	24.8 [0.0125, 470]	31.0 [0.0125, 440]	17.0 [0.0125, 470]
Missing, No. (%)	469 (2.8)	405 (2.3)	44 (3.6)	32 (3.6)	176 (1.9)	234 (3.3)	1360 (2.6)
Years since quit,b median (range)	14.0 [1.00, 63.0]	14.0 [1.00, 67.0]	16.0 [1.00, 54.0]	17.0 [1.00, 57.0]	17.0 [1.00, 60.0]	20.0 [1.00, 61.0]	16.0 [1.00, 67.0]
Total eligible for screening , No. (%)
USPSTF 2021 guidelinesc	5213 (31.7)	7430 (42.8)	486 (39.8)	391 (43.5)	4634 (49.8)	3619 (50.6)	21 773 (41.5)
ACS 2023 recommendationsd	6135 (37.3)	8286 (47.7)	576 (47.2)	482 (53.7)	5626 (60.4)	4642 (64.9)	25 747 (49.1)
USPSTFSmoke Duratione	10 793 (65.5)	12 986 (74.8)	764 (62.6)	572 (63.7)	5785 (62.1)	4176 (58.3)	35 076 (66.9)
PLCOm2012 (1.51% threshold)f	4988 (30.3)	7179 (41.3)	369 (30.2)	307 (34.2)	4114 (44.2)	3441 (48.1)	20 398 (38.9)
PLCOm2012No Race (1.51% threshold)g	3761 (22.8)	5563 (32.0)	396 (32.4)	322 (35.9)	4183 (44.9)	3470 (48.5)	17 695 (33.8)
PLCOm2012Update (1.3% threshold)h	6383 (38.8)	8702 (50.1)	414 (33.9)	346 (38.5)	4493 (48.2)	3756 (52.5)	24 094 (46.0)
LCRAT (1.4% threshold)i	6723 (40.8)	9646 (55.5)	328 (26.9)	298 (33.2)	4404 (47.3)	3771 (52.7)	25 170 (48.0)
LCDRAT (0.8% threshold)j	7392 (44.9)	10 542 (60.7)	317 (26.0)	308 (34.3)	4295 (46.1)	3831 (53.5)	26 685 (50.9)

Abbreviations: ACS = American Cancer Society; LCDRAT = Lung Cancer Death Risk Assessment Tool; LCRAT = Lung Cancer Risk Assessment Tool; PLCOm2012 = Prostate, Lung, Colorectal and Ovarian Cancer 2012 risk prediction model; USPSTF = US Preventive Services Task Force.

^aAmong lung cancer cases.

^bAmong persons who formerly smoked.

^cUSPSTF 2021 guidelines target individuals aged 50-80 years with a minimum 20-pack-year history who either currently smoke or quit within the previous 15 years.⁷

^dACS 2023 recommendations are a modified version of the USPSTF 2021 guidelines that remove the quit time criterion.¹⁷

^eUSPSTFSmoke Duration is a modified version of the USPSTF 2021 guidelines that uses years smoked instead of pack years.¹⁶

^fPLCOm2012 predicts 6-year lung cancer incidence risk.²¹

^gPLCOm2012No Race is a reparametrized version of the PLCOm2012 model that removes race as a predictor.³⁰

^hPLCOm2012Update is an updated version of the original PLCOm2012 model with race coefficient recalibrated using the Multiethnic Cohort.³¹

ⁱLCRAT predicts 5-year lung cancer incidence risk.^20,38

^jLCDRAT predicts 5-year lung cancer death risk.^20,38

Model performance: percentage eligible for screening, sensitivity, and specificity

Across all screening strategies, the overall percentage of participants eligible for screening, regardless of lung cancer status, ranged from 34% (PLCOm2012_NoRace) to 67% (USPSTF_{SmokeDuration}) (Table 1). This percentage varied across race and sex. Black/African American women had the lowest percentage of participants eligible (range = 23%-45%), whereas White men had the highest percentage (range = 48%-65%) across all strategies, except USPSTF_{SmokeDuration} (Table 1, Figure 1). When USPSTF_{SmokeDuration} was used as the screening strategy, the percentage eligible was reversed, with White men having the lowest eligibility (58%).

Figure 1.

Sensitivity of risk factor–based and risk model–based screening eligibility strategies, by race and sex. USPSTF 2021 guidelines target individuals aged 50-80 years with a minimum 20-packyear history who either currently smoke or quit within the previous 15 years.⁷ ACS 2023 recommendations are a modified version of the USPSTF 2021 guidelines that removes the quit time criterion.¹⁷ USPSTF_{SmokeDuration} is a modified version of the USPSTF 2021 guidelines that uses years smoked instead of pack-years.¹⁶ USPSTF_RaceSpecific is a modified version of the USPSTF 2021 guidelines with race-specific pack-year criteria based on our previous recommendations for addressing racial disparities in screening eligibility.¹⁵ Black/African American individuals are eligible for screening with a minimum 13-pack-year smoking history; White individuals are eligible for screening with a minimum of 20-pack-year smoking history. PLCOm2012 predicts 6-year lung cancer incidence risk.²¹ PLCOm2012_NoRace is a reparametrized version of the PLCOm2012 model that removes race as a predictor.³⁰ PLCOm2012_Update is an updated version of the original PLCOm2012 model with the race coefficient recalibrated using the Multiethnic Cohort..³¹ PLCOm2012_RaceSpecific is the original PLCOm2012 model modified to allow participants to be eligible for screening using race-specific risk thresholds, similar to our previous recommendation.¹⁵ Black/African American individuals are eligible for screening based on a 0.87% risk threshold; White individuals are eligible based on a 1.51% risk threshold. LCRAT predicts 5-year lung cancer incidence risk.^20,38 LCDRAT predicts 5-year lung cancer death risk.^20,38 ACS = American Cancer Society; LCDRAT = Lung Cancer Death Risk Assessment tool; LCRAT = Lung Cancer Risk Assessment Tool; PLCOm2012 = Prostate, Lung, Colorectal and Ovarian Cancer model 2012; USPSTF = US Preventive Service Task Force.

Risk factor–based strategies

Among the 3 risk factor–based strategies, USPSTF_{SmokeDuration} had the highest overall sensitivity (83%) and the lowest overall specificity (33%) and showed no evidence of racial disparity. In contrast, the USPSTF 2021 guidelines yielded a 62% sensitivity and a 58% specificity, but with substantial racial disparities (Table S2). The ACS recommendations also showed racial disparities, with higher sensitivity and lower specificity among White and Multiracial/Other participants compared with Black/African American participants. In fact, the racial disparity was greatest with ACS, which resulted in a sensitivity 18 percentage points (95% CI = 15 to 22) lower for Black/African American participants compared with their White counterparts (Figure 2, Table S2). Across all risk factor–based strategies, there was no evidence of sex disparities. Racial disparities, however, were present in select race-sex subgroups (Figure 1). For example, USPSTF_{SmokeDuration} showed no evidence of racial disparity among women. Both USPSTF 2021 and ACS, however, showed racial disparities for Black/African American women vs White women who were diagnosed with lung cancer. Estimates of the differential sensitivity were 23 (95% CI = 18 to 28) to 25 (95% CI = 20 to 30) percentage points lower for Black/African American women. The finding for men was more complicated. The USPSTF 2021 and ACS yielded a sensitivity that was 7 (95% CI = 3 to 12) and 14 (95% CI = 10 to 19) percentage points lower, respectively, for Black/African American men compared with White men. However, USPSTF_{SmokeDuration} reversed this finding, with a sensitivity that was 11 (95% CI = 7 to 15) percentage points higher for Black/African American men compared with White men.

Figure 2.

Sensitivity and specificity of risk factor–based screening strategies in Black/African American and White individuals. Black/African American individuals are in black, and White individuals are in blue. (A) USPSTF 2021 guidelines, (B) ACS 2023 recommendations, (C) USPSTF_{SmokeDuration}, and (D) USPSTF_RaceSpecific were used to determine lung screening eligibility. Pack-years = 20 used for A and B; years smoked = 20 used for C; race-specific pack-years in D (Black/African American = 13; White = 20). Arrows indicate racial disparity. Numerical summaries are provided in Table S2. ACS = American Cancer Society; USPSTF = US Preventive Services Task Force.

Risk model–based strategies

The ROC curves of all the risk model–based strategies had similar shape and areas under the curve, implying essentially equivalent diagnostic potential (Figure 3, Figure S1, Figure S2). Each model operated at a different point on the ROC curve, however, determined by a strategy-specific risk threshold, resulting in mixed performance across race groups. The LCDRAT, LCRAT, and PLCOm2012_Update performed similarly when considered overall (sensitivity = 74%, 73%, and 72%; specificity = 48%, 51%, and 53%, respectively), whereas PLCOm2012 and PLCOm2012_NoRace had lower sensitivities and higher specificities (sensitivity = 64% and 58%; specificity = 60% and 65%, respectively) (Tables S3 and S4). When compared across races, however, all models except LCDRAT found more White individuals with lung cancer eligible for screening than Black/African American individuals with lung cancer (Tables S3 and S4).

Figure 3.

Receiver operating characteristic curves for risk model–based screening strategies in Black/African American and White individuals. Black/African American individuals appear in black, and White individuals appear in blue. (A) PLCOm2012,  (B) PLCOm2012_NoRace, (C) PLCOm2012_Update, and (D) PLCOm2012_RaceSpecific were used to determine lung screening eligibility. A screening threshold of 1.51% was used for A and B; 1.3% was used for C; race-specific thresholds were used in D (Black/African American = 0.87%, White = 1.51%). Arrows indicate racial disparity. Numerical summaries are provided in Table S3. AUC = area under the curve; PLCOm2012 = Prostate, Lung, Colorectal and Ovarian Cancer 2012 risk prediction model.

Similar to the risk factor–based strategies, there was no evidence of disparities across sex using any of the risk model–based strategies. Racial disparities were present, however, in varying degrees across select race-sex groups (Figure 1 and Table S3). For example, PLCOm2012 resulted in a sensitivity that was 22 (95% CI = 17 to 27) percentage points lower for Black/African American women compared with White women. In comparison, the LCRAT had a 15 (95% CI = 11 to 20) percentage point sensitivity difference, whereas the LCDRAT had a 9 (95% CI = 4 to 13) percentage point sensitivity difference (both lower for Black/African American women). Modifications to the PLCOm2012 model affected the race-sex disparities, with models developed from more representative data exhibiting less algorithmic bias. PLCOm2012_NoRace reduced sensitivity among Black/African American participants and resulted in a large disparity between White and Black/African American women (33 [95% CI = 29 to 38] percentage point difference). In contrast, PLCOm2012_Update improved sensitivity and resulted in a smaller race-sex disparity than PLCOm2012 (15 [95% CI = 10 to 20] percentage point difference). Similar patterns were observed among men (Table S3).

Mitigating algorithmic bias

Uniform screening thresholds

Here, a fixed risk threshold was used to determine eligibility. For the PLCOm2012 models, we compared each model’s performance using its original validation threshold (Table S1) vs a higher threshold of 2.0% selected to match the Ontario Lung Screening Program’s eligibility threshold.^37,38 Compared with the original validation thresholds, using higher risk thresholds decreased sensitivity and increased specificity for all subgroups, leading to poorer screening performance with no meaningful changes in disparities (Table S5, Figure S1).

Impact of race-aware fine-tuning

Here, race-specific pack-year and risk thresholds chosen to yield equal sensitivity across race population groups were used to determine eligibility. The USPSTF_RaceSpecific guidelines yielded a sensitivity of 71% and a specificity of 50% for both Black/African American and White individuals (Figures 1 and 2D, Table S2). The improved sensitivity among Black/African American individuals resulted in a smaller race-sex disparity between Black/African American and White women when compared with the USPSTF 2021 guidelines (5 [95% CI = 1 to 10] vs 23 [95% CI = 18 to 28] percentage point difference). PLCOm2012_RaceSpecific also eliminated White–Black/African American disparities and yielded a substantially smaller race-sex disparity between Black/African American and White women (Figure 1, Figure S2, Table S3). A race-specific threshold modification chosen to have equal specificity instead of sensitivity yielded similar results (Table S5).

A blended screening strategy

Here, we combined risk factor–based (eg, USPSTF 2021) and risk model–based (eg, PLCOm2012) strategies to evaluate the effect of augmenting simple criteria with complex risk models (Tables S6-S9). This approach was informed by the National Comprehensive Cancer Network’s recommendation to use risk models to quantify risk among high-risk individuals who do not meet eligibility using the USPSTF criteria.¹⁸

We found that if both the risk factor and risk model screening strategies exhibited racial disparities, their combination did not improve disparities. If both screening strategies exhibited racial equity, their combination generally was equitable between White and Black/African American individuals. The scenario was more complicated when 1 strategy exhibited racial disparities and the other did not. Often the performance depended on the extent of algorithmic bias and racial disparity. For example, combining USPSTF 2021 (exhibits racial disparities) with PLCOm2012_RaceSpecific (exhibits racial equity) yielded a sensitivity of 76% and a specificity of 47%, with an 8 [95% CI = 5 to 11] percentage point sensitivity difference between White and Black/African American individuals. Combining USPSTF_{SmokeDuration} (exhibits racial equity) with PLCOm2012 (exhibits racial disparities), however, yielded a sensitivity of 89%, with no racial disparity between Black/African American and White individuals but a low overall specificity of 27%.

Discussion

We found that the performance among several lung cancer screening eligibility strategies varied substantially by race and sex. Most strategies showed evidence of algorithmic bias, identifying fewer Black/African American participants who would later develop lung cancer as eligible for screening vs their White counterparts. Racial eligibility disparities were amplified at the intersection of race and sex, with Black/African American women having the lowest screening eligibility. Our approach, which tailored eligibility thresholds by race to account for algorithmic bias, yielded the best sensitivity-specificity trade-off with the smallest racial disparities.

A screening strategy must delicately balance several factors, including sensitivity, specificity, ease of application, disease prevalence, cost of misclassification, and equity across subgroups. Compared with the USPSTF guidelines, risk model–based strategies such as PLCOm2012 include nuanced measures of smoking exposure and other risk factors, which can improve performance metrics and minimize misclassification. These risk models, however, can introduce racial disparities when training data are not generalizable. Our findings suggest that simply including race as a main effect in a regression model does not sufficiently resolve racial disparities in screening eligibility. Instead, fine-tuning the screening algorithm in a manner that accommodates imbalances in critical subgroups is better able to achieve performance equity across population subgroups.

Our results confirm prior work conducted in the SCCS that using smoke duration rather than pack-years yielded high sensitivity and minimal racial disparities compared with the USPSTF 2021 guidelines.¹⁶ Yet, USPSTF_{SmokeDuration} also yielded a specificity (33%) that appears too low for it to be a viable national screening strategy. This rate is nearly half the specificity of the current guidelines and incorrectly selected 67% of participants who did not develop cancer for screening in this high-risk population. An effective screening strategy should maximize test accuracy and minimize risks among the screened population.^39-41 However, USPSTF_{SmokeDuration} improves racial equity by effectively screening everybody at risk along with nearly everybody who was not at risk. For people who never develop lung cancer, annual low-dose computed tomography may lead to false-positive results, which may require further diagnostic workup with potential complications. In a study of real-world lung screening practices, Rendle et al.⁴² found that more than one-third of patients completing a screening low-dose computed tomography scan had findings that needed subsequent imaging or further diagnostic workup, including invasive procedures. Given the potential downstream risks involved with low-dose computed tomography screening, careful consideration must be given to whether this cost of increased eligibility misclassification is a justifiable price to pay to improve racial equity.

In its guidance on assessing bias in clinical prediction models, the USPSTF recognizes use of different decision thresholds as a means to ensure fair distribution of services.^5,34 Our findings show that approaches such as USPSTF_RaceSpecific and PLCOm2012_RaceSpecific (which use different eligibility thresholds by race to ensure equal diagnostic performance across race) yielded the best sensitivity-specificity trade-off with the smallest racial disparities compared with the current USPSTF guidelines. They also yielded smaller race-sex disparities. This finding aligns with our prior conclusions that there is likely no one-size-fits-all threshold that will serve all populations at risk of lung cancer equitably.^15,43 This work provides an approach for the scientific community to evaluate whether patterns of algorithmic bias exist and to resolve disparities in screening eligibility until alternative approaches to equitable screening eligibility are identified.

The USPSTF also recommends that careful consideration be given to potential negative effects of bias-mitigating strategies.⁵ We must decide as a scientific community what the optimal risk-benefit trade-off is to identify the most individuals who are at risk for lung cancer for screening equitably. The optimal trade-off should be achieved within all relevant subgroups as opposed to across the entire high-risk population. Policy makers must also pay close attention to how risk thresholds are used to determine eligibility to avoid unintentionally perpetuating disparities. Should the current guidelines remain, eligibility disparities will remain. Adjusting risk thresholds can address current bias in algorithms and improve the identification of high-risk cases. Until improved risk models are developed, race-aware fine-tuning of eligibility thresholds is likely needed to achieve equitable screening eligibility.

We acknowledge a few limitations. First, our Multiracial/Other group combined participants from different backgrounds with known differences in lung cancer risk. Thus, our performance estimates for this group represent an average effect that may not apply to any 1 subgroup. Future studies should endeavor to disaggregate race and ethnicity data to target screening resources where they are most needed. Our analysis was focused on addressing the differential performance of risk models between White and Black/African American participants. Given that lung cancer risk distributions and smoking behaviors vary across geographic region and populations, patterns of algorithmic bias should be evaluated for other racial and ethnic groups. In addition, differences in downstream factors may influence equity in screening outcomes. Future research can build upon our findings by examining how disparities in access, adherence, and follow-up care interact with eligibility guidelines to affect disparities in screening outcomes. Next, we used the last observation carried forward method to impute follow-up observations that were missing that could lead to an overestimation of performance. A sensitivity analysis using multiple imputation, however, did not change our conclusions. Finally, we did not assess patient perspectives on race-specific eligibility thresholds for screening. Future studies should explore how patients perceive and respond to such approaches to ensure that efforts to enhance equity are aligned with patient values and preferences.

Our study is strengthened by more than a decade of follow-up with ascertainment of detailed smoking behaviors and other known lung cancer risk factors as well as identification of incident lung cancers. The SCCS represents a population of predominantly Black/African American individuals with a high smoking prevalence recruited from federally qualified health centers across the Southern United States, portraying a population with low socioeconomic status and a high prevalence of smoking for whom lung cancer screening is particularly relevant.

These findings suggest that eligibility thresholds tailored by race yield sensitivity/specificity trade-offs with the smallest race-sex disparities. Using race-aware approaches may be necessary to achieve more equitable lung screening eligibility.

Author contributions

Adoma Manful (Conceptualization, Data curation, Formal analysis, Methodology, Software, Validation, Visualization, Writing—original draft, Writing—review & editing), Sarah F. Mercaldo (Methodology, Software, Writing—review & editing), Jeffrey Blume (Conceptualization, Funding acquisition, Methodology, Resources, Supervision, Validation, Visualization, Writing—review & editing), and Melinda C. Aldrich (Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Supervision, Visualization, Writing—review & editing)

Supplementary material

Supplementary material is available at JNCI: Journal of the National Cancer Institute online.

Funding

This work was supported by theNational Cancer Institute of the National Institutes of Health under award No. R01CA251758 to M.C.A. and J.D.B. and No. U01CA202979. The SCCS data collection was performed by the Survey and Biospecimen Shared Resource, which is supported in part by the Vanderbilt-Ingram Cancer Center (P30 CA68485). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Conflicts of interest

The authors have no conflicts of interest to declare.

Data availability

The data underlying this article were provided by the SCCS team and are available to qualified researchers upon request (https://www.southerncommunitystudy.org/for-researchers.html).