Artificial intelligence and lung cancer treatment decision: agreement with recommendation of multidisciplinary tumor board

Abstract

Background

IBM Watson for Oncology (WFO) is a cognitive computing system helping physicians quickly identify key information in a patient’s medical record, surface relevant evidence, and explore treatment options. This study assessed the possibility of using WFO for clinical treatment in lung cancer patients.

Methods

We evaluated the level of agreement between WFO and multidisciplinary team (MDT) for lung cancer. From January to December 2018, newly diagnosed lung cancer cases in Chonnam National University Hwasun Hospital were retrospectively examined using WFO version 18.4 according to four treatment categories (surgery, radiotherapy, chemoradiotherapy, and palliative care). Treatment recommendations were considered concordant if the MDT recommendations were designated ‘recommended’ by WFO. Concordance between MDT and WFO was analyzed by Cohen’s kappa value.

Results

In total, 405 (male 340, female 65) cases with different histology (adenocarcinoma 157, squamous cell carcinoma 132, small cell carcinoma 94, others 22 cases) were enrolled. Concordance between MDT and WFO occurred in 92.4% (k=0.881, P<0.001) of all cases, and concordance differed according to clinical stages. The strength of agreement was very good in stage IV non-small cell lung carcinoma (NSCLC) (100%, k=1.000) and extensive disease small cell lung carcinoma (SCLC) (100%, k=1.000). In stage I NSCLC, the agreement strength was good (92.4%, k=0.855). The concordance was moderate in stage III NSCLC (80.8%, k=0.622) and relatively low in stage II NSCLC (83.3%, k=0.556) and limited disease SCLC (84.6%, k=0.435). There were discordant cases in surgery (7/57, 12.3%), radiotherapy (2/12, 16.7%), and chemoradiotherapy (15/129, 11.6%), but no discordance in metastatic disease patients.

Conclusions

Treatment recommendations made by WFO and MDT were highly concordant for lung cancer cases especially in metastatic stage. However, WFO was just an assisting tool in stage I–III NSCLC and limited disease SCLC; so, patient-doctor relationship and shared decision making may be more important in this stage.

Introduction

Although recent progress in precision medicine applies to cancer therapy, lung cancer remains the leading cause of cancer-related mortality worldwide, including East Asia (1). There are various treatment strategies according to stage, tumor site, histology and genetic alteration. Many hospitals constitute multidisciplinary teams (MDT) consisting of oncologists, surgeons, radiologists, radiation oncologists, pathologists and palliative care specialists to make optimal decisions for patients with lung cancer (2). In Korea, the number of newly diagnosed lung cancer cases is steadily increasing (3-5), decreasing the time that doctors can dedicate to learning. On the other hand, new drugs, medical data, papers and guidelines for lung cancer are rapidly growing. Thus, even oncologists who are experts in a specialized field cannot master all available knowledge. Recent advances help oncologists quickly identify key information in a patient’s medical record, surface relevant evidence and explore treatment options (6-8). Artificial intelligence (AI) is a general term which denotes the use of a computer system to model intelligent processes so as to lower human intervention (9). AI systems in oncology acquires knowledge from large medical datasets guidelines. They, then, use computational reasoning approaches to apply it to a specific case, and generate insights for clinicians (10).

Watson for Oncology (WFO, IBM Watson Health, Cambridge, MA) is a clinical decision-support system (CDSS) for treatment of lung cancer, breast cancer and prostate cancer at Memorial Sloan Kettering Cancer Center (MSKCC) in March 2012 (11-15). WFO was introduced in South Korea in 2017, and has been assisting clinicians at several cancer centers. WFO stores and indexes literature, protocols, and patient charts. It learns from test cases and all the information input is verified by the experts from MSKCC. Moreover, WFO data are updated to the latest cutting-edge information every 1 to 2 months. When we input a case that is not supported by WFO, the system does not recommend a treatment plan. In case of lung cancer, patients with isolated metastatic tumors and those with driver mutations whose cancer progresses during metastatic therapy are not yet supported by WFO. With its growing popularity, many clinicians are questioning whether WFO is suitable for cancer management. Cancer patients also worry about receiving treatment recommendations from WFO. The complexity of lung cancer treatment tends to decrease the consistency of recommendations from WFO in comparison to those from the MDT. There are only a few published studies investigating the reliability of WFO in Manipal Comprehensive Cancer Center (11) and China (16,17).

We conducted this study to assess the agreement between WFO and MDT at a single cancer center in South Korea. The objective of this study was to determine the level of recommended treatment concordance in lung cancer cases.

Methods

Study design and population

We conducted a retrospective study on 405 cases of lung cancer and compared the degree of agreement of initial treatment recommendations for lung cancer cases, between WFO and an MDT at the Chonnam National University Hwasun Hospital (CNUHH) in South Korea.

The inclusion criteria for this study were as follows: (I) patients with primary lung cancer; (II) those who were admitted between January 2018 and December 2018; (III) those who did not receive antitumor treatment. The exclusion criteria were as follows: (I) secondary lung cancer which has a distant site of origin (II) those who had previously received antitumor treatment. Patients who received only confirmed diagnosis and not any antitumor treatment were also included.

Multidisciplinary tumor board for lung cancer

The MDT for lung cancer was composed of pulmonary oncologists, radiation oncologists, thoracic surgeons, neurosurgeons, radiologists, pathologists and nuclear medicine doctors. After new cases of lung cancer were confirmed, their medical records were reviewed at MDT conferences twice every week. They classified the treatment decision in four categories: surgery, radiotherapy, concurrent chemoradiotherapy (CCRT), and medical treatments including chemotherapy, targeted therapy, immunotherapy.

WFO

WFO supports doctors in making lung cancer treatment decisions using a curated body of knowledge including text from more than 300 medical journals and textbooks, MSKCC treatment guidelines, and literature hand-selected by MSKCC experts (11). For supported cases, WFO analyzed medical records and provided treatment plans in three categorized groups with corresponding labels: ‘recommended treatments’ with green label (as a strong base of evidence), ‘for consideration’ with amber label (as appropriate alternatives based on their clinical judgment), and ‘not recommended’ with green label (as specific contraindications or strong evidence against their use). WFO version 18.4 was used in this study.

Data collection & statistical analysis

Patient’s data were abstracted from electronic medical records and entered manually into WFO by one trained fellow. The MDT tumor board had previously reviewed and recommended treatment regimens for all new cases in 2018. WFO analyzed the same cases along with their clinical information in 2019. WFO and the physicians who ran the cases were blinded to the treatment recommendations that had been made by the MDT.

Treatment recommendations were considered concordant if the tumor board’s recommendation corresponded to the ‘recommended’ categories. If WFO suggested two or more plans as ‘recommended’ categories and MDT’s plan corresponded to one of them, it was regarded as concordant. We analyzed concordance rate by cancer stage and histology.

Statistical analysis was performed with IBM SPSS statistics version 23.0 (SPSS, Inc., an IBM Company, Chicago, IL, USA), and differences with P values of less than 0.05 were considered statistically significant. Cancer characteristics included patient age, sex, Eastern Cooperative Oncology Group (ECOG) performance status (18), cancer histology and stage. Concordance rate between MDT and WFO was expressed as percent agreement and Cohen's kappa value. A logistic regression model was estimated with odds ratio and a 95% confidence interval (CI).

Results

Baseline characteristics

We could collect 463 cases who were presumed to have lung cancer and were admitted to CNUHH for definitive diagnosis from January to December 2018. After excluding 58 patients according to the eligibility criteria, a total of 405 patients were matched up with the inclusion criteria and received treatment decisions from the MDT (Figure 1).

Figure 1.

Flow diagram of the patient selection process. CNUHH, Chonnam National University Hwasun Hospital; MDT, multidisciplinary team.

The median age was 71 years, and 83.9% of patients were men (Table 1). Most of the patients showed ECOG performance status 0 to 1, but 11.4% and 1.9% of patients had statuses of 3 and 4, respectively. The histology of 289 cases (71.4%) were non-small cell lung cancer (NSCLC), composed of adenocarcinoma (157 cases; 38.8%) and squamous cell carcinoma (132 cases; 32.6%). Among the rest, 94 cases (23.2%) were small cell lung carcinoma (SCLC).

Table 1. Baseline characteristics of patients.

Characteristics	Classify	N=405
Age, median year [range]	–	71 [37–88]
Gender	Male/Female	340 (83.9)/65 (16.1)
ECOG performance status, (n, %)	0/1/2/3	94 (46.8)/80 (39.8)/23 (11.4)/4 (1.9)
Histology, (n, %)	Adenocarcinoma	157 (38.8)
	Squamous cell carcinoma	132 (32.6)
	Small cell carcinoma	94 (23.2)
	The others	22 (5.4)
Stage, (n, %)	Stage I NSCLC	31 (7.7)
	Stage II NSCLC	38 (9.4)
	Stage III NSCLC	95 (23.5)
	Stage IV NSCLC	158 (39.0)
	Limited stage SCLC	39 (9.6)
	Extensive stage SCLC	44 (10.8)

ECOG, Eastern Cooperative Oncology Group; NSCLC, non-small cell lung cancer; SCLC, small cell lung cancer.

Concordance between WFO and MDT

Overall treatment concordance between MDT and WFO was 92.4% (kappa value =0.881; P<0.001). The concordance rates according to histology were 94.9% (k=0.900), 90.2% (k=0.857) and 97.9% (k=0.934) for adenocarcinoma, squamous cell carcinoma and small cell lung cancer, respectively.

When concordance was analyzed by both stage and histology, metastatic cases were found to have 100% concordance between MDT and WFO’s decisions (Table 2). High concordances were shown in stage I NSCLC (92.4%, k=0.855), stage IV NSCLC (100%, k=1.000) and extensive disease SCLC (100%, k=1.000). But the concordance rates were 83.3% (k=0.556), 80.8% (k=0.622), and 84.6% (k=0.435) in stage II NSCLC, stage III NSCLC, and limited disease SCLC, respectively.

Table 2. Comparison between WFO and MDT recommendation by stage.

Stage and histology	MDT suggestion				Subtotal	Concordance rate (%)	Kappa value
	Surgery	RT	CCRT	Medical
Stage I NSCLC						90.3	0.757
WFO suggestion
Surgery	21	1	0	0	22
RT	2	7	0	0	9
Subtotal	23	8	0	0	31
Stage II NSCLC						86.8	0.726
WFO suggestion
Surgery	24	1	3	0	28
RT	0	3	0	0	3
CCRT	0	1	6	0	7
Subtotal	24	5	9	0	38
Stage III NSCLC						85.3	0.498
WFO suggestion
Surgery	5	1	1	0	7
RT	3	3	73	6	85
CCRT	0	0	0	3	3
Subtotal	8	4	74	9	95
Stage IV NSCLC						100.0	1.000
WFO suggestion
Medical	0	0	0	158	158
Subtotal	0	0	0	158	158
Limited SCLC						94.9	0.645
WFO Suggestion
CCRT	1	0	35	1	37
Medical	0	0	0	2	2
Subtotal	1	0	35	3	39
Extensive SCLC						100.0	1.000
WFO suggestion
Medical	0	0	0	44	44
Subtotal	0	0	0	44	44

The results of logistic regression of concordance as a function of ECOG score, histology combined with stage are presented in Table 3. An odds ratio greater than 1 indicates greater odds of concordance, equal to 1 suggests equal odds, and less than one indicates lesser odds. Except metastatic cases in which treatment decisions between WFO and the MDT were perfectly coincident, there was no statistically significant factor that affected concordance between WFO and MDT’s decisions.

Table 3. Logistic regression model of concordance between WFO and the MDT.

Characteristics	Case number & percentage	Odds ratio (95% CI)	P value
Age
<60 years (reference)	74 (18.3)	1.000
>60 years	331 (81.7)	0.897 (0.125–3.254)	0.589
Gender
Male (reference)	340 (83.9)	1.000
Female	65 (16.1)	1.135 (0.268–2.987)	0.698
ECOG score
0 (reference)	94 (46.8)	1.000
1	80 (39.8)	1.255 (0.444–3.548)	0.669
2	23 (11.4)	0.497 (0.179–1.378)	0.497
3	4 (1.9)	0.559 (0.064–4.927)	0.601
Histology with stage
Stage I NSCLC (reference)	31 (7.7)	1.000
Stage II NSCLC	38 (9.4)	0.707 (0.155–3.225)	0.654
Stage III NSCLC	95 (23.5)	0.620 (0.166–2.318)	0.477
Stage IV NSCLC	158 (39.0)	173,086,590.3	0.995
Limited SCLC	39 (9.6)	1.982 (0.310–12.673)	0.470
Extensive SCLC	44 (10.8)	173,086,590.3	0.998

Discordances were found in the case of surgery (7/57, 12.3%) and concurrent chemoradiotherapy (CCRT) (15/129, 11.6%). But there was no discordance in patients who were recommended for medical treatment due to metastatic stage. The main reason for not performing surgery in early stage NSCLC was because of underlying disease, old age, poor ECOG score (Table 4). The reason for not performing radiotherapy was patient’s preference for surgery. There were 14 cases of discordance in stage III NSCLC, and various reasons existed among physicians and patients.

Table 4. Discordant cases between WFO and the MDT recommendation by stage.

Stage	Discordant case, No.	WFO suggestion	MDT suggestion	Causes of discordance [No. of cases]
Stage I NSCLC	1	Surgery	RT	Combined IPF [1]
	2	RT	Surgery	Patient’s preference [2]Ϯ
Stage II NSCLC	4	Surgery	CCRT	Risk for pneumonectomy [2]ф Inadequate surgical candidate [1]
			RT	Inadequate surgical candidate [1]
	1	CCRT	RT	Risk for chemotherapy [1]
Stage III NSCLC	2	Surgery	CCRT	Risk for pneumonectomy [1]ф
			RT	Inadequate surgical candidate [1]
	12	CCRT	Surgery	Patient’s preference [3]Ϯ
			RT	Risk for chemotherapy [3]
			Medical treatment	Inadequate CCRT candidate [3] Suspected advanced stage [2]ǂ For targeted therapy [1]
Limited SCLC	2	CCRT	Surgery	For tissue diagnosis [1]Ұ
			Medical treatment	Inadequate CCRT candidate [1]

^Ϯ, necessity to try surgical resection such as young age patient or patient's will; ^ф, mass site was considered to need pneumonectomy during surgery, which will be an impossible burden on patients; ^ǂ, include pleural effusion which had not be proven by cytology and too extensive size of mass; ^Ұ, clinically suspected to be SCLC, but not proven by biopsy. NSCLC, non-small cell lung cancer; SCLC, small cell lung cancer; WFO, Watson for Oncology; MDT, multidisciplinary team; IPF, idiopathic pulmonary fibrosis.

Discussion

This retrospective study demonstrated that an AI-based CDSS trained by experts in the United States could also be used feasibly in Korea. An overall concordance rate of 92.4% was present between MDT and WFO in patients with lung cancer. The strength of agreement was very high in metastatic stage regardless of histology. Treatment recommendations were also highly concordant in stage I NSCLC. However, it was relatively low in stage II–III NSCLC and limited disease SCLC. The concordance rate was only 80.8% in stage III NSCLC. Discordances were most frequently found in case of WFO’s decision for CCRT which was changed to other treatment options by physicians or patients.

Replacing the doctor with an intelligent medical robot is an interesting concept in science fiction. AI in health care is nowadays exist close at hand (7). Machine learning means that the computer learns to perform tasks by analyzing data rather than requiring specific programming instructions from humans, so that they generate their own decision-making algorithms (6). Machine learning has the potential to be extremely useful in medicine, particularly in the interpretation of medical images such as computed tomography and histopathological slides (6,19). It may increase the speed and consistency of diagnosis, but it may also exacerbate overdiagnosis (6). Recently, Xu et al. demonstrated that deep learning can integrate imaging scans at multiple time-points to improve clinical outcome predictions (8). Although AI-based noninvasive radiomics have a great deal of promise, it also has inherent limitations, particularly when it comes to diagnosing early-stage cancer because there is no single right answer to the question (6,20).

In metastatic stage, the treatment option for lung cancer is relatively simple (chemotherapy, targeted therapy or immunotherapy). Most treatments are decided by a physician according to the patients’ performance status. But decision making in earlier stages of lung cancer is more complex because of many patient-related factors associated with co-morbidity, insurance, socioeconomic state or preference. An MDT system and shared decision making is very important in this situation. Despite the overall discordant rate between WFO and MDT (7.6%), all discordances were observed in non-metastatic stages. For example, surgery could not be performed because of several factors like idiopathic pulmonary fibrosis or tumor location which requires pneumonectomy. Inversely, the MDT made decisions to conduct surgery in some cases where the WFO recommended non-surgical treatment. The reason behind adopting surgery was because of individual patient circumstances such as when the patient is young. Because WFO could not reflect patient statuses in detail, some discordance did occur between WFO and MDT in non-metastatic stage, in our study.

Although AI technology continues to evolve, there are additional reasons for the discordance between WFO and MDT across countries. For example, national medical guidelines, ethnic differences in cancer patients, national licensing of recommended drugs or treatments, or compliance with insurance coverage and screening standards are thought to affect the discordance rate among different countries (12).

This study has several limitations. First, this was a small sample-sized retrospective observational study conducted at a single cancer center. Second, the decision between MDT and WFO was not made simultaneously. The MDT made decisions each time the patient had been diagnosed with lung cancer, and the patient’s medical records were input into WFO in the following year. Hence, some changes of treatment guidelines that were not reflected in the study might have occurred. Third, some elderly patients who visited the emergency room but did not want to get biopsies done were not included in this study. Our results could not be applied to patients without tissue confirmation. Finally, we could not analyze detailed treatment options like surgery type, radiation dose or fraction, and regimen of chemotherapy. Because the WFO suggests a lot of chemotherapeutic regimens at once and the regimen changes very quickly, it was difficult to determine whether they were concordant between the WFO and the MDT. Several new drugs that WFO suggested were difficult to use because they were not covered by government insurance.

Conclusions

In conclusion, treatment decisions made by WFO exhibited a high degree of agreement with those of the MDT tumor board, and the concordance varied by stage. AI-based CDSS is expected to play an assistive role, particularly in the metastatic lung cancer stage with less complex treatment options. However, patient-doctor relationships and shared decision making may be more important in non-metastatic lung cancer because of the complexity to reach at an appropriate decision. Further study is warranted to overcome this gray area for current machine learning algorithms.

Supplementary

The article’s supplementary files as

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by independent Institutional Review Board (IRB) of Chonnam National University Hwasun Hospital (IRB approval number: CNUHH-2019-195) and individual consent for this retrospective analysis was waived.