The National Surgical Quality Improvement Program risk calculator does not adequately stratify risk for patients with clinical stage I non–small cell lung cancer

Abstract

Objective

The study objective was to validate the National Surgical Quality Improvement Program (NSQIP) Risk Calculator in stratifying risk estimates for patients who received surgery or stereotactic body radiation therapy for clinical stage I non–small cell lung cancer.

Methods

A retrospective analysis of patients with clinical stage I non–small cell lung cancer undergoing surgery (N = 279) or stereotactic body radiation therapy (N = 206) from 2009 to 2012 was performed. NSQIP complication risk estimates were calculated for both surgical and stereotactic body radiation therapy cases using the NSQIP Surgical Risk Calculator. NSQIP complication risk estimates were compared as continuous variables and by quartile ranges.

Results

Compared with patients undergoing video-assisted thoracoscopic surgery wedge resection, patients receiving stereotactic body radiation therapy were older, had larger tumors, had lower forced expiratory volume (FEV₁) in 1 second and diffusing capacity of the lungs (DLCO) for carbon monoxide values, had higher American Society of Anesthesiologists scores, had higher rates of dyspnea, and had higher NSQIP serious complication risk estimates (all P<.05). Compared with patients undergoing video-assisted thoracoscopic surgery lobectomy, patients receiving stereotactic body radiation therapy had similar disparities, along with higher Adult Comorbidity Evaluation-27 (ACE) scores comorbidity scores, higher rates of cardiac comorbidities, and worse functional status (all P<.05). Variables associated with receiving stereotactic body radiation therapy treatment, rather than wedge resection, included increasing age, higher Adult Comorbidity Evaluation (ACE)-27 comorbidity score, dyspnea status, and decreasing FEV₁ in 1 second and DLCO for carbon monoxide, but NSQIP serious complication risk score. In addition, surgical patients’ actual serious complication rate (16.6% vs 8.8%) and pneumonia rate (6.0% vs 3.2%) were significantly higher than the NSQIP risk calculator predicted (all P < .05).

Conclusions

The National Surgical Quality Improvement Program risk calculator does not effectively classify or stratify risk in patients with stage I non–small cell lung cancer. Continued efforts are needed to assess risk in this population and develop more tailored treatment decision aids.

Graphical abstract

Range of NSQIP risk scores for patients undergoing VATS lobectomy (blue) and patients receiving SBRT (green). NSQIP, National Surgical Quality Improvement Program; VATS, video-assisted thoracoscopic surgery; SBRT, stereotactic body radiation therapy.

Approximately 15% of patients with non–small cell lung cancer (NSCLC) are diagnosed with stage I disease.¹ Although lobectomy with lymphadenectomy represents the gold standard of treatment for early-stage NSCLC and offers the best long-term survival in medically fit patients, many patients present with multiple comorbidities and poor pulmonary function. In these situations, surgeons may elect to proceed with sublobar resection (segmentectomy or wedge resections) or refer patients for stereotactic body radiation therapy (SBRT). Allocation of the appropriate therapy for patients with early-stage NSCLC remains a clinical and subjective challenge in high-risk but potentially operable patients. To date, 3 randomized controlled trials have attempted to address this clinical conundrum, but all have closed secondary to poor accrual.^2–4

In 2013, the American College of Surgeons used the National Surgical Quality Improvement Program (NSQIP) database to create an online surgical risk calculator, using inputs from approximately 1.4 million patients at 393 hospitals for more than 1500 Current Procedural Terminology codes.⁵ The use of the NSQIP Surgical Risk Calculator has been encouraged by the American College of Surgeons to improve both the informed consent process and shared decision making between the physician and patient.^6,7 However, the applicability and performance of the NSQIP Surgical Risk Calculator in a population with clinical stage I NSCLC are unknown at this time. The specific aims of this study were to retrospectively validate the NSQIP Surgical Risk Calculator’s ability to 1) predict 30-day outcomes and mortality among patients with stage I NSCLC who received surgical resection and 2) to clinically discriminate between surgical candidates and patients who were deemed inoperable and referred for SBRT.

MATERIALS AND METHODS

After approval by the institutional review board of Washington University in St Louis, our prospectively maintained clinical stage I NSCLC database was reviewed to identify patients undergoing surgical resection or SBRT from January 2009 to December 2012. This database is a collaborative effort of the Division of Cardiothoracic Surgery and Department of Radiation Oncology at our institution. This analysis started in 2009 to provide a consistent volume of patients receiving SBRT and ended in 2012 with the most recent data available from the Department of Radiation Oncology. Patients with a history of lung cancer were excluded. Data abstracted included patient demographics, tumor characteristics (clinical stage, tumor size on 2-dimensional chest computed tomography imaging), patient comorbidities (by both individual disease classification and Adult Comorbidity Evaluation-27 [ACE] score), and pulmonary function tests (forced expiratory volume in 1 second [FEV₁] and diffusing capacity of the lungs for carbon monoxide [DLCO]). For patients undergoing surgical resection, information regarding surgical approach (thoracotomy vs video-assisted thoracoscopic surgery [VATS]), extent of resection (wedge resection, segmentectomy, lobectomy, bilobectomy, or pneumonectomy), postoperative complications, and 30-day mortality was obtained. On review of the database, 49 patients (9%) had a missing ACE score, 7 patients (1%) had missing body mass index, and 51 patients (9.4%) had missing FEV₁ percent predicted values.

By using the American College of Surgeon’s NSQIP Surgical Risk Calculator website (http://riskcalculator.facs.org/), the Current Procedural Terminology code of the procedure they had performed (VATS wedge: 32666, open wedge: 32605, VATS segmentectomy: 32669, open segmentectomy: 32484, VATS lobectomy: 32663, open lobectomy: 32480, open bilobectomy: 32482, or pneumonectomy 32440) was entered, along with the appropriate patient characteristics. Patients who received SBRT had their risk estimates separately calculated as if they were being considered for VATS lobectomy or VATS wedge resection. These 2 surgical approaches were chosen because they would provide the largest comparator groups in the surgical population and were the most likely procedures an operable high-risk surgical patient may actually receive. The NSQIP Surgical Risk Calculator requires 22 patient characteristics to be entered, as summarized in Table E1. Patients receiving SBRT were excluded if they were deemed medically fit for surgery but declined operative intervention for other reasons. After patient data are entered into the online calculator, estimated risk percentages are reported for serious complication, any complication, pneumonia, cardiac complication, surgical site infection, urinary tract infection, venous thromboembolism, renal failure, reoperation, and death.

Descriptive statistics for normally distributed continuous variables were expressed as mean ± standard deviation with independent-sample t tests. If continuous variables did not demonstrate a normal distribution, logarithmic transformation was used and geometric means were obtained for analysis. Chi-square tests to analyze categoric variables were performed. The 95% confidence intervals (CIs) were calculated for the complication rate of patients undergoing surgery using the equation, $p-1.96\sqrt{[p(1-p)/n]}$, $p+1.96\sqrt{[p(1-p)/n]}$ where p = proportion of patients with the complication of interest and n = number of subjects undergoing surgery. This was done to facilitate comparison with the mean values with 95% CIs obtained from the NSQIP calculated complication scores. If the 95% CIs did not overlap, then values were interpreted as being significantly different. For comparing the distribution of NSQIP serious complication risk estimate values for surgical versus SBRT treatment groups, comparative histograms were plotted (both as a continuous variable and in quartiles). Backward stepwise logistic regression was performed on candidate variables from univariate analysis (if P < .05) to identify factors associated with a higher likelihood of receiving SBRT. Receiver operating curves (ROCs) were created to analyze the performance of the NSQIP serious complication risk score in predicting SBRT treatment. All analyses were performed in SPSS (IBM, Version 22.0, Armonk, NY).

RESULTS

National Surgical Quality Improvement Program Surgical Risk Calculator Performance in Patients With Stage I Non–Small Cell Lung Cancer Receiving Pulmonary Resection

From 2009 to 2012, 277 patients underwent surgical resection for clinical stage I NSCLC, and 195 patients received SBRT. Of the surgical patients, 205 (74.5%) received a lobectomy, 53 (19.3%) received a wedge, 11 (4.0%) received a segmentectomy, 4 (1.5%) received a pneumonectomy (because the central anatomy was not amenable to a sleeve resection), and 2 (0.7%) received a bilobectomy (because of tumors involving the bronchus intermedius). A total of 243 patients (84.1%) receiving surgery underwent a VATS approach, 45 patients (15.6%) had a thoracotomy, and 1 patient (0.3%) received a sternotomy. Compared with patients receiving a wedge resection, those receiving a lobectomy (VATS or thoracotomy) had lower ACE comorbidity scores (57.3% had a score of 0 or 1 vs 32.7% in the wedge resection group, P = .002), had a larger preoperative lesion size (2.4 ± 0.9 cm vs 1.8 ± 0.9 cm, P < .001), had a higher FEV₁ percent predicted (82.7% ± 20% vs 71.6% ± 20%, P < .001) and DLCO percent predicted (73.6% ± 24% vs 61.4% ± 21%, P = .002), were less likely to have a history of congestive heart failure (0.5% vs 5.7%, P = .007), were more likely to have a lower American Society of Anesthesiologists (ASA) class (21% were class 2 vs 7.5% of wedge resection cases, P = .02), were less likely to have dyspnea with moderate exertion (24.9% vs 45.3%, P = .004), and were less likely to have had a cardiac event (12.7% vs 24.5%, P = .03).

The actual complication rates of surgical patients were compared with the mean values of the calculated NSQIP estimated rates, with 95% CIs (Table 1). The rates of serious complication and pneumonia were significantly higher than the NSQIP ranges predicted for this patient population. NSQIP predicted risk scores for events including “any complication,” cardiac complication, surgical site infection, urinary tract infection, venous thromboembolism, renal failure, return to the operating room, and death demonstrated overlap with the actual complication rates and calculated 95% CIs for surgical patients, and therefore did not demonstrate a significant difference. When the analysis was limited to those surgical patients receiving a VATS procedure, the rate of serious complications in that population remained significantly higher than the NSQIP estimates (14%, 95% CI, 10–18 vs the NSQIP estimate of 8.6%, 95% CI, 8.3–8.9).

TABLE 1.

Actual surgical complication rates in 277 patients undergoing pulmonary resection from 2009 to 2012 compared with the mean National Surgical Quality Improvement Program predicted complication rates for that population

Complication type	Actual surgical complication rate (with calculated 95% CI), n = 277	NSQIP predicted complication rate (with 95% CI)
Serious complication*	16.6 (12.2–21.0)	8.8 (8.4–9.1)
Any complication†	17.3 (12.8–21.8)	13.0 (12.6–13.5)
Pneumonia	6.0 (4.0–9.0)	3.2 (3.0–3.5)
Cardiac complication‡	1.4 (0.03–2.9)	0.7 (0.6–0.7)
Surgical site infection	1.8 (0–4.0)	1.6 (1.5–1.6)
Urinary tract infection	4.0 (1.0–6.0)	1.8 (1.7–1.9)
Venous thromboembolism	4.0 (1.0–6.0)	1.0 (0.97–1.1)
Renal failure	1.0 (0–2.3)	0.6 (0.6–0.7)
Return to operating room	4.0 (2.0–7.0)	2.8 (2.5–2.9)
Death	1.0 (0–3.0)	1.8 (1.6–1.9)

The 95% CIs were calculated for the actual surgical population to compare intervals with the NSQIP mean population ranges. Complication rates were determined to be statistically significant if their 95% CIs demonstrated no overlap. CI, Confidence interval; NSQIP, National Surgical Quality Improvement Program.

^*NSQIP defines serious complication as cardiac arrest, myocardial infarction, pneumonia, progressive renal insufficiency, acute renal failure, pulmonary embolism, deep vein thrombosis, return to the operating room, deep incisional surgical site infection, organ space surgical site infection, systemic sepsis, unplanned intubation, urinary tract infection, or wound disruption.

^†The NSQIP any complication category includes all those listed as serious complications, with the additions of superficial incisional surgical site infection, ventilator use more than 48 hours, and stroke.

^‡NSQIP defines cardiac complication as cardiac arrest or myocardial infarction (myocardial infarction based on electrocardiogram changes, new troponin elevation, or physician diagnosis).

National Surgical Quality Improvement Program Surgical Risk Estimates for Video-Assisted Thoracoscopic Surgery Wedge Resection in Patients With Stage I Non–Small Cell Lung Cancer

Univariate analysis of patients with clinical stage I NSCLC receiving VATS wedge (n = 51) or SBRT (n = 195) is shown in Table 2. Compared with patients undergoing a VATS wedge resection, patients receiving SBRT were older, had a larger preoperative lesion size, had a lower FEV₁ percent predicted and DLCO percent predicted, a higher ASA classification, were more likely to have dyspnea with moderate activity or at rest, and had a higher NSQIP estimated serious complication risk score. Factors associated with allocation to SBRT compared with VATS wedge resection on stepwise logistic regression included increasing age (by year, odds ratio [OR], 1.09; 95% CI, 1.04–1.15; P = .001) and increasing preoperative lesion size (by centimeter, OR, 2.56; 95% CI, 1.45–4.51; P = .001). Decreased likelihood of allocation to SBRT included increasing FEV1 percent predicted value (by single % change, OR, 0.96; 95% CI, 0.94–0.99; P = .001).

TABLE 2.

Univariate analysis of patient characteristics comparing patients who received video-assisted thoracoscopic surgery wedge resection with patients who received stereotactic body radiation therapy from 2009 to 2012

Variable	Surgery VATS wedge resection (n = 51)	SBRT (if considered for VATS wedge resection, n = 195)	P value
Age, y	66.4 ± 10.9	74.1 ± 8.4	<.001
Male gender	24 (47.1%)	85 (43.6%)	.66
Race (Caucasian)	42 (82.4%)	168 (86.2%)	.08
BMI (kg/m2)	27.9 ± 6.5	26.7 ± 6.9	.24
ACE comorbidity score
0–1	15 (30.6%)	72 (32.1%)	.003
2	22 (44.9%)	51 (22.8%)
3	12 (24.5%)	101 (45.1%)
Preoperative lesion size (cm)	1.8 ± 0.9	2.4 ± 0.8	<.001
FEV1 (% predicted)	71.9% ± 20.4%	56.9% ± 21.6%	<.001
DLCO (% predicted)	62.0% ± 21.7%	51.5% ± 20.5%	.006
Congestive heart failure	3 (5.9%)	32 (16.4%)	.06
Coronary artery disease	14 (27.5%)	69 (35.4%)	.29
Peripheral vascular disease	4 (7.8%)	35 (17.9%)	.08
Hypertension requiring medication*	34 (66.7%)	137 (70.3%)	.62
History of MI	5 (9.8%)	21 (10.8%)	.84
ASA class*
1	0	0	.55
2	3 (6.8%)	11 (4.3%)
3	34 (77.3%)	213 (83.9%)
4	7 (15.9%)	30 (11.8%)
Steroids for chronic condition*	3 (5.9%)	17 (8.7%)	.51
History of smoking in past year*	25 (49.0%)	96 (49.2%)	.98
Dialysis*	1 (2.0%)	2 (1.0%)	.59
Functional status*
Independent	51 (100%)	182 (93.3%)	.06
Partially or totally dependent		13 (6.7%)
Dyspnea status*
None	28 (54.9%)	53 (27.2%)	<.001
Moderate activity or rest	23 (45.1%)	142 (72.8%)
Previous cardiac event*	12 (23.5%)	60 (30.8%)	.31
Diabetes status*
None	43 (84.3%)	159 (81.5%)	.86
Oral medications	6 (11.8%)	25 (12.8%)
Insulin	2 (3.9%)	11 (5.6%)
CHF exacerbation in past 30 d*	1 (2.0%)	2 (1.0%)	.59
NSQIP risk of serious complication (%)	6.7 ± 2.6	7.8 ± 3.1	.02
NSQIP risk of 30-d mortality (%)	1.1% ± 0.9%	1.8% ± 1.5%	<.001

Variables marked with an asterisk (*) indicate patient information that is required by the NSQIP Surgical Risk Calculator. VATS, Video-assisted thoracoscopic surgery; SBRT, stereotactic body radiation therapy; BMI, body mass index; ACE, Adult Comorbidity Evaluation-27; FEV₁, forced expiratory volume in 1 second; DLCO, diffusing capacity of the lungs for carbon monoxide; MI, myocardial infarction; ASA, American Society of Anesthesiologists; CHF, congestive heart failure; NSQIP, National Surgical Quality Improvement Program.

To further evaluate the characteristics of the NSQIP Surgical Risk Calculator in quantifying risk among patients with clinical stage I NSCLC, the NSQIP risk estimates for serious complication for both VATS wedge resection cases and SBRT cases were divided into quartiles (Figure 1, A). Although the patients receiving SBRT did not undergo surgery, their score was calculated as if they were being considered for a VATS wedge resection. Although approximately half (45%) of the patients undergoing VATS wedge resection had a serious complication risk estimate of less than 5.6%, the SBRT patient estimates were more evenly distributed among the quartiles. Although the mean values of VATS wedge resection versus SBRT serious complication risk estimates were significantly different on univariate analysis (6.7% vs 7.8%, respectively, P<.001), the actual range of these scores for patients receiving SBRT does not provide substantial clinical dichotomization. This can also be seen when NSQIP serious complication risk estimates are plotted as a continuous variable for the VATS wedge resection cases and SBRT cases (Figure 2, A). Substantial overlap can be seen over the range of scores in these 2 populations. Finally, receiver operating curves (ROCs) were created to examine how effective the NSQIP serious complication risk estimate was in predicting SBRT use (Figure E1, A). For VATS wedge resections, the area under the curve (AUC) value was 0.65 (95% CI, 0.56–0.74), indicating a poor test for predicting likelihood of SBRT use with increasing risk estimate values.

FIGURE 1.

A, Frequency of VATS wedge resection surgical cases (blue) and SBRT cases (green) by quartile of NSQIP serious complication risk estimates. Quartiles were generated from the entire range of NSQIP serious complication risk estimate scores for both the VATS wedge surgical cases and the SBRT cases (as if they were being considered for a VATS wedge resection). The difference between the distributions of these quartiles by treatment type was significant (P < .001). B, Frequency of VATS lobectomy surgical cases (blue) and SBRT (green) by quartile of NSQIP serious complication risk estimate. Quartiles were generated from the entire range of NSQIP serious complication risk estimate scores for both the VATS lobectomy cases and the SBRT cases (as if they were being considered for a VATS lobectomy). The difference between the distributions of these quartiles by treatment type was significant (P < .001). NSQIP, National Surgical Quality Improvement Program; NSCLC, non–small cell lung cancer; VATS, video-assisted thoracoscopic surgery; SBRT, stereotactic body radiation therapy.

FIGURE 2.

A, Frequency plot of VATS wedge resection (blue) and SBRT (green) by NSQIP serious complication risk estimate. Despite a significant difference in the mean risk estimate values on univariate analysis, the plot demonstrates substantial overlap for VATS wedge resection and SBRT cases. B, Frequency plot of surgical resection (blue) and SBRT (green) by NSQIP serious complication risk estimate for VATS lobectomy cases and SBRT cases (had they received a VATS lobectomy). Despite a significant difference in the mean risk estimate values on univariate analysis, the frequency plot demonstrates substantial overlap for VATS lobectomy and SBRT cases. NSQIP, National Surgical Quality Improvement Program; VATS, video-assisted thoracoscopic surgery; SBRT, stereotactic body radiation therapy.

National Surgical Quality Improvement Program Surgical Risk Estimates for Video-Assisted Thoracoscopic Surgery Lobectomy in Patients With Stage I Non–Small Cell Lung Cancer

Comparison of patient characteristics between patients receiving a VATS lobectomy (n = 178) and the 195 patients which received SBRT (this time with risk estimates calculated on the prospect of receiving a VATS lobectomy) is shown in Table 3. Compared with surgical patients receiving a VATS lobectomy, those receiving SBRT were older; had higher ACE comorbidity scores; had lower percent predicted FEV₁ and DLCO; were more likely to have a history of congestive heart failure, coronary artery disease, peripheral vascular disease, and myocardial infarction; were more likely to have a higher ASA classification, use steroids for a chronic medical condition, and have a previous cardiac event; were less likely to have an independent functional status; and were more likely to have dyspnea with moderate exertion or rest, and a higher NSQIP risk of serious complication risk estimate value.

TABLE 3.

Univariate analysis of patient characteristics comparing patients who received video-assisted thoracoscopic surgery lobectomy with patients who received stereotactic body radiation from 2009 to 2012

Variable	Surgery (VATS lobectomy, n = 178)	SBRT (if considered for VATS lobectomy, n = 195)	P value
Age, y	66.6 ± 10.1	74.1 ± 8.4	<.001
Male gender	76 (42.7%)	85 (43.6%)	.86
Race (Caucasian)	152 (85.4%)	168 (86.2%)	.18
BMI (kg/m2)	27.6 ± 7.7	26.7 ± 6.9	.21
ACE comorbidity score
0–1	101 (59.1%)	72 (32.1%)	<.001
2	45 (26.3%)	51 (22.8%)
3	25 (14.6%)	101 (45.1%)
Preoperative lesion size (cm)	2.4 ± 0.9	2.4 ±0.8	.53
FEV1 (% predicted)	82.8% ± 19.6%	56.9% ± 21.6%	<.001
DLCO (% predicted)	74.3% ± 23.4%	51.5% ± 20.5%	<.001
Congestive heart failure	1 (0.6%)	32 (16.4%)	<.001
Coronary artery disease	30 (16.9%)	69 (35.4%)	<.001
Peripheral vascular disease	17 (9.6%)	35 (17.9%)	.02
Hypertension requiring medication*	114 (64.0%)	137 (70.3%)	.20
History of MI	7 (4.0%)	21 (10.8%)	.01
ASA Class*
1	0		<.001
2	31 (20.7%)	11 (4.3%)
3	111 (74.0%)	213 (83.9%)
4	8 (5.3%)	30 (11.8%)
Steroids for chronic condition*	4 (2.2%)	17 (8.7%)	.007
History of smoking in past year*	79 (44.4%)	96 (49.2%)	.35
Dialysis*	2 (1.1%)	2 (1.0%)	.93
Functional status*
Independent	178 (100%)	182 (93.3%)	<.001
Partially or totally dependent		13 (6.7%)
Dyspnea status*
None	133 (74.7%)	53 (27.2%)	<.001
Moderate activity or rest	45 (25.3%)	142 (72.8%)
Previous cardiac event*	22 (12.4%)	60 (30.8%)	<.001
Diabetes status*
None	161 (90.4%)	159 (81.5%)	.05
Oral medications	11 (6.2%)	25 (12.8%)
Insulin	6 (3.4%)	11 (5.6%)
CHF exacerbation in past 30 d*	0 (0%)	2 (1.0%)	.18
NSQIP risk of serious complication (%)	9.3% ± 2.5%	12.4% ± 4.6%	<.001
NSQIP risk of 30-d mortality (%)	0.93% ± 0.84%	2.1% ± 1.7%	<.001

Variables marked with an asterisk (*) indicate patient information that is required by the NSQIP Surgical Risk Calculator. VATS, Video-assisted thoracoscopic surgery; SBRT, stereotactic body radiation therapy; BMI, body mass index; ACE, Adult Comorbidity Evaluation-27; FEV₁, forced expiratory volume in 1 second; DLCO, diffusing capacity of the lungs for carbon monoxide; MI, myocardial infarction; ASA, American Society of Anesthesiologists; CHF, congestive heart failure; NSQIP, National Surgical Quality Improvement Program.

On logistic regression, factors independently associated with allocation to SBRT in comparison with VATS lobectomy included increasing age (per year, OR, 1.08; 95% CI, 1.04–1.12; P < .001), an ACE comorbidity score of 3 (reference: 0–1, OR, 2.44; 95% CI, 1.10–5.39; P = .03), a history of congestive heart failure (OR, 13.5; 95% CI, 1.36–134.9; P = .03), the presence of dyspnea with exertion (OR, 6.65; 95% CI, 3.48–12.7; P < .001), and increasing NSQIP serious complication risk estimate (per percent, OR, 1.19; 95% CI, 1.04–1.37; P = .01). Increasing DLCO was associated with a decreased likelihood of receiving SBRT (per percent, 0.96, 95% CI, 0.95–0.98; P < .001).

Again, quartiles of NSQIP serious complication risk estimates were generated for the range of VATS lobectomy scores (including surgical patients who received a VATS lobectomy and patients who received SBRT as if they were being considered for a VATS lobectomy). Figure 1, B demonstrates a more linear relationship between the risk estimate and the treatment received as the NSQIP serious complication risk estimate increases, the percentage of patients receiving SBRT increases, and the percentage of patients receiving VATS lobectomy decreases. However, when plotted as continuous variable (Figure 2, B), substantial overlap is again seen across the spectrum of serious complication risk estimates for these 2 groups. Figure E1, B shows the ROC curve for the NSQIP serious complication risk estimate in predicting treatment allocation to VATS lobectomy or SBRT, with an improved AUC value of 0.78 (95% CI, 0.73–0.82), indicating a fair test.

DISCUSSION

Overall, these results demonstrate that patients undergoing VATS wedge resection or lobectomy for early-stage NSCLC have significantly lower NSQIP serious complication risk estimates than patients receiving SBRT. Although the NSQIP Surgical Risk Calculator provided a fair level of discrimination between VATS lobectomy and SBRT on ROC analysis, it was a poor model for differentiating between VATS wedge resection and SBRT. Unfortunately, it is this later population of the highest risk surgical patients (for whom a lobectomy is not a surgical option) where risk models and decision aids are needed the most. In addition, the NSQIP Surgical Risk Calculator underestimated the total serious complication risk and pneumonia risk for patients receiving any type of pulmonary resection at our institution.

Previous definitions of the “high-risk” surgical patient have been proposed, but have proven a challenge to validate. The “high-risk” criteria for pulmonary resection in the American College of Surgery Oncology Group z4032 trial and z4099 included FEV₁ or DLCO 50% or less, age 75 years or more, a diagnosis of pulmonary hypertension, left heart failure, resting hypoxia or hypercarbia, and elevated dyspnea scale scores, as defined by expert consensus.⁸ Our group at Washington University has previously reported that application of these criteria in our surgical population demonstrated no difference in postoperative complications or 30-day mortality between the high-risk cohort and non–high-risk patients.⁸ In contrast, the NSQIP Surgical Risk Calculator seems to underestimate the overall rate of serious complications and pneumonia in the same population of surgical patients. This finding of NSQIP underestimating the actual complication rate was also found at another high-volume academic thoracic surgery center that compared lobectomy complication rates in the Society of Thoracic Surgeons General Thoracic Database (STS GTDB) with the NSQIP database.⁹ Allen and colleagues⁹ described that although the STS-GTDB (which tracked 100% of operations for 2012) described a 10.9% pneumonia rate for lobectomy at the Mayo Clinic, the NSQIP database (which recorded 19.3% of their surgeries the same year) estimated a 5.9% pneumonia rate. When examining all thoracic surgeries during 2012 (including pulmonary resections, esophageal resections, and laparoscopic Nissen procedures), the STS-GTDB found an overall postoperative event rate of 30.1%, whereas NSQIP again underestimated this prevalence with a 17.2% complication rate.⁹

Previous pulmonary resection risk models created from the STS GTSD are certainly comprehensive in scope and well powered (drawing from >18,000 lung cancer resections at 111 centers).¹⁰ However, these analyses are based on a range of clinical T and N groups. Predictors of mortality in the STS GTDB model included pneumonectomy, bilobectomy, increasing ASA or Zubrod performance score, history of renal dysfunction, induction chemoradiation therapy, a history of steroid use, age, an urgent procedure, male gender, FEV₁, and body mass index.¹⁰ Although this was an important and landmark review¹⁰ in understanding risk factors in patients undergoing pulmonary resection, this model has not been adapted into an interface for clinical practice to calculate risk scores for patients. Furthermore, not all variables listed in the STS GTDB model would be applicable to a patient with clinical stage I NSCLC: urgent procedure, induction chemoradiation therapy, or pneumonectomy. The fact that these variables are included in the STS GTDB analysis means that other factors that would apply to patients with stage I disease would be weighted differently on logistic regression, and the strengths of association would be altered accordingly.

Several variables that were different between the surgical and SBRT cases on univariate analysis were similar to predictors of mortality from the STS GTDB risk analysis: older age, lower FEV₁ percent predicted, ASA class, and chronic steroid use.¹⁰ This suggests that clinicians are currently considering these variables as part of their decision-making process. Our analysis found that increasing DLCO was independently associated with receiving VATS lobectomy when comparing risk estimates with patients receiving SBRT (had they been considered for VATS lobectomy), similar to another study demonstrating an association of low DLCO with a higher rate of pulmonary morbidity and perioperative mortality.¹¹ In the current NSQIP Surgical Risk Calculator, severe chronic obstructive pulmonary disease is defined as an FEV₁ percent predicted of less than 75%. However, thoracic surgeons have historically used a preoperative predicted FEV₁ value of 40% as an operative (or at least a high risk) threshold for lobectomies; therefore, there are many surgical cases and SBRT cases with FEV₁ greater than 40%, but less than 75%, and this may be a cause of substantial overlap in both the lobectomy and wedge resection estimates. In addition, the minimum FEV₁ percent predicted value for patients receiving SBRT in our series was 10% and the minimum DLCO value was 5.2%. Therefore, the NSQIP Surgical Risk Calculator would weigh the patient with an FEV₁ percent predicted of 10% or 65% the same, despite 10% or 65%, despite the fact that these values represent clearly different pulmonary reserve values for surgery. The NSQIP calculator’s lack of discriminatory ability regarding the spectrum of pulmonary function may limit its ability to predict outcomes in patients undergoing thoracic surgery. Although pulmonary function is just one component of complex risk assessment, capturing the full spectrum of pulmonary function values among these patients may help provide more nuanced risk scores for assessing patients for operative versus nonoperative therapy.

Study Limitations

There are limitations to this analysis. Although the temporal constraints of our study limits our ability to make a robust independent model predicting SBRT use, it is adequately powered to detect the performance of the NSQIP Surgical Risk Calculator in our population, in both the surgical and SBRT cases. Because the patients receiving SBRT did not undergo surgery, we calculated their risk estimates as if they were being considered for a VATS lobectomy or a VATS wedge resection. We chose to examine VATS lobectomy scores among the patients receiving SBRT because this represents the preferred approach if deemed fit for surgery, but is likely not medically feasible for many of these patients. This is confirmed by the improved ROC AUC results (0.78) for the VATS lobectomy NSQIP serious complication risk estimates in predicting SBRT use among patients compared with the ROC AUC for wedge scores (0.65). However, we also considered the VATS wedge resection score among the patients receiving SBRT because this likely represents an alternative to SBRT in high-risk but potentially operable patients who would not be able to tolerate a lobectomy. This is supported by recent studies that have suggested no difference in disease-free survival in both matched and unmatched comparisons of wedge resection versus SBRT.^4,12 In fact, determining the relative survival advantage of SBRT versus sublobar resection is a primary focus of many practitioners involved in the care of high-risk patients with early-stage NSCLC. With this in mind, there are other clinical subtleties, such as tumor location (central vs peripheral parenchymal position), anticipated tumor margins, and patient preferences, that would need to be included in a decision model of treatment allocation.

CONCLUSIONS

Counseling the high-risk but operable patient with clinical stage I NSCLC in regard to lobectomy, sublobar resection, or SBRT is challenging for both the clinician and the patient. For these individuals, there is an extensive assessment of the patient’s medical history, consideration of competing medical and social demands, and an understanding of the patient’s risk tolerance level and overall clinical picture. Even a perfect model cannot capture all of these factors. We believe that a model tailored to patients with clinical stage I needs to serve as both an estimator of operative risks and a patient decision aid for surgery versus SBRT, especially with projected increases in the number of early-stage lung cancers as a result of increased lung cancer screening efforts. At this time, our analysis suggests that the NSQIP Surgical Risk Calculator likely does not profile the risk of a patient with lung cancer closely enough to reliably dichotomize surgical and inoperable SBRT cases (especially when patients are being considered for a wedge resection) or accurately estimate a surgical patient’s risk of serious complications. Although robust in its collection methods across a spectrum of surgical procedures, the NSQIP risk calculator’s algorithm is derived from surgical patients only. To create a risk calculator that can help clinicians prospectively examine the risk of patients with early-stage NSCLC (for clinical trial eligibility, tumor board discussions, and routine consultations), multidisciplinary and multi-institutional efforts will be necessary to capture the characteristics and outcomes of patients receiving SBRT.

Central Message.

At this time, the NSQIP Surgical Risk Calculator does not accurately estimate risk among patients with clinical stage I NSCLC.

Perspective.

NSQIP risk estimate scores were retrospectively calculated for patients with early-stage NSCLC undergoing surgical resection or SBRT. The NSQIP risk calculator underestimated serious complication risk in operative patients and did not adequately stratify risk values between surgical and SBRT cases to assist in prospective clinical decision making, especially for sublobar resection.

Abbreviations and Acronyms

ACE

Adult Comorbidity Evaluation-27

ASA

American Society of Anesthesiologists

AUC

area under the curve

CI

confidence interval

DLCO

diffusing capacity of the lungs for carbon monoxide

FEV₁

forced expiratory volume in 1 second

NSCLC

non–small cell lung cancer

NSQIP

National Surgical Quality Improvement Program

OR

odds ratio

ROC

receiver operating curve

SBRT

stereotactic body radiation therapy

STS GTDB

Society of Thoracic Surgeons General Thoracic Database

VATS

video-assisted thoracoscopic surgery