Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift

Abstract

On November 30, 2022, OpenAI enabled public access to ChatGPT, a next-generation artificial intelligence with a highly sophisticated ability to write, solve coding issues, and answer questions. This communication draws attention to the prospect that ChatGPT and its successors will become important virtual assistants to patients and health-care providers. In our assessments, ranging from answering basic fact-based questions to responding to complex clinical questions, ChatGPT demonstrated a remarkable ability to formulate interpretable responses, which appeared to minimize the likelihood of alarm compared with Google’s feature snippet. Arguably, the ChatGPT use case presents an urgent need for regulators and health-care professionals to be involved in developing standards for minimum quality and to raise patient awareness of current limitations of emerging artificial intelligence assistants. This commentary aims to raise awareness at the tipping point of a paradigm shift.

On November 30, 2022, OpenAI enabled public access to ChatGPT, a next-generation artificial intelligence (AI) optimized for conversational and interactive dialog (1). ChatGPT has caused widespread interest for its sophisticated ability to write stories, essays, solve coding issues, and answer questions across a broad range of topics including politics, engineering, and medicine (2,3). In addition, there have been discussions to ChatGPT impacting research, learning, and clinical practice workflows (4). This communication would like to draw attention to prospects that ChatGPT and its successors will become virtual assistants to patients, for easy access to information and answers.

At a basic level, ChatGPT can be used to generate answers to some of the most Googled facts about cancer, such as the following:

1. What are the most common cancer types?

2. What exactly is cancer?

Supplementary Table 1 (available online) provides the Google response and 5 example ChatGPT responses to these fact-based questions. For these questions, ChatGPT provided succinct responses similar in quality to the feature snippet (colloquially “answer”) of Google—chiefly that 1) breast, prostate, and lung cancer are the most common cancers, and 2) cancer is a disease where body cells grow uncontrolled. However, for question 1, ChatGPT has generally provided contextualization that cancer risks can be different according to patient characteristics—nuances not in Google’s feature snippet. For question 2, though still concise, ChatGPT demonstrated variability in supplementary information (eg, discussions on causes, treatments, and symptoms), presumably to make the response conversational. Further differences are that ChatGPT does not return weblinks, and ChatGPT will output different answers (potentially profoundly so) on requestioning, even with the same prompt. In contrast, Google’s feature snippet has a webpage link (in this case to National Cancer Institute webpages), and within-day searching was not associated with answer changes (searches occurred on December 13, 2022, Australia), noting that Google’s feature snippet will not necessarily come from such a reputable resource, and the snippet can change over time.

However, the disruptive nature of ChatGPT is not its ability to answer simple questions. Rather, it’s the ability to formulate responses and advice to complex questions and conversations, which is well beyond any other (widely used) search technology. In this paper, we present ChatGPT responses to 3 clinically relevant example questions covering diagnosis, prognosis, and treatment (Table 1):

Table 1.

Google and ChatGPT answers to some example questions from patients with cancer

Questions	Is coughing a sign of lung cancer?	What is the prognosis of metastatic pancreatic cancer?	Does pembrolizumab cause fever, and should I go to the hospital?
Google feature snippet with associated link	Returns about 7 510 000 weblink results.	Returns about 29 600 000 weblink results.	Returns about 119 000 weblink results.
	Featured snippet: “The main symptoms of lung cancer include a cough that does not go away after 3 weeks; a long-standing cough that gets worse; chest infections that keep coming back.” Link: https://www.nhs.uk/conditions/lung-cancer/symptoms/	Featured snippet: “Stage IV pancreatic cancer has a five-year survival rate of 1%. The average patient diagnosed with late-stage pancreatic cancer will live for about 1 year after diagnosis.” Link: https://www.hopkinsmedicine.org/health/conditions-and-diseases/pancreatic-cancer/pancreatic-cancer-prognosis	No featured snippet returned for this question. First link: https://www.keytruda.com/side-effects/
ChatGPT response 1	“Coughing can sometimes be a sign of lung cancer, but it can also be a symptom of many other conditions. Some other possible causes of a cough include asthma, bronchitis, the common cold, and pneumonia. If you have a persistent cough that does not go away, it is important to see a doctor for a proper diagnosis. They can determine the cause of your cough and recommend the appropriate treatment.”	“Metastatic pancreatic cancer is a very serious and difficult to treat form of the disease. The prognosis for people with metastatic pancreatic cancer is generally poor, with most people surviving less than six months after being diagnosed. However, it is important to remember that each person is different, and some people may be able to live longer with the disease. Treatment options, such as chemotherapy and radiation, can help to extend life expectancy and improve quality of life. It is important to work with a healthcare team to develop a treatment plan that is tailored to your individual needs.”	“I’m sorry, but I am not able to browse the internet and therefore do not have access to information about the effects of pembrolizumab. I would strongly recommend speaking with a healthcare professional if you are experiencing fever and are concerned about your health. They will be able to provide you with personalized medical advice based on your specific situation.”
ChatGPT response 2	“Coughing can sometimes be a sign of lung cancer, but it can also be a sign of many other things, such as a cold or flu, bronchitis, or even just something stuck in your throat. It’s important to remember that only a doctor can tell you for sure what is causing your cough. If you have a persistent cough that won’t go away, it’s important to see a doctor to determine the cause and get the appropriate treatment.”	“The prognosis for metastatic pancreatic cancer is generally poor. This type of cancer has a tendency to spread quickly and is often difficult to treat. In most cases, the cancer has already spread to other parts of the body by the time it is diagnosed, making it difficult to completely remove the cancer. The average life expectancy for people with this type of cancer is about six months, although some people may live longer with treatment. However, treatment options for metastatic pancreatic cancer are limited and may not be effective in all cases. It is important to talk with a doctor about the specific prognosis for an individual’s case.”	“Pembrolizumab is a type of cancer treatment known as an immune checkpoint inhibitor. It can cause side effects such as fever, but it is important to discuss any concerns with your doctor. If you are experiencing fever or other severe symptoms, you should contact your doctor immediately and follow their advice. They will be able to provide you with personalized recommendations based on your individual situation. In some cases, it may be necessary to go to the hospital for further evaluation and treatment.”
ChatGPT response 3	“Coughing can be a sign of many different things, including lung cancer. However, it is important to note that coughing is a common symptom of many different conditions, not just lung cancer. If you are experiencing a cough, it is important to see a doctor to determine the underlying cause. They will be able to provide a proper diagnosis and recommend the best course of treatment.”	“Metastatic pancreatic cancer, which is cancer that has spread from the pancreas to other parts of the body, is typically considered to be advanced and difficult to treat. The prognosis for metastatic pancreatic cancer is generally not very good, and the average life expectancy for people with this condition is only a few months. However, the exact prognosis will vary depending on a variety of factors, such as the stage of the cancer, the person’s overall health and age, and the treatments that are available. In some cases, treatment may be able to slow the progression of the cancer and improve the person’s quality of life, but a cure is often not possible.”	“Pembrolizumab is a type of medication used to treat certain types of cancer. It is not known to cause fever as a common side effect, but it is always important to consult with your doctor if you have any concerns about your health. If you are experiencing a fever, it would be a good idea to contact your doctor or seek medical attention to determine the cause and receive appropriate treatment. It is always better to be safe than sorry, so if you are unsure, it is best to consult with a healthcare professional.”
ChatGPT response 4	“Coughing can sometimes be a sign of lung cancer, but it can also be caused by many other things, such as a cold or the flu. It’s important to remember that only a doctor can diagnose the cause of a cough. If you’re experiencing a cough that lasts for a long time or is accompanied by other symptoms, such as chest pain or difficulty breathing, it’s a good idea to see a doctor for a proper evaluation. They can determine the cause of your cough and recommend the appropriate treatment.”	“Metastatic pancreatic cancer is a very serious condition with a poor prognosis. The survival rate for people with this type of cancer is very low, and treatment options are often limited. In general, the outlook for people with metastatic pancreatic cancer is not good, and the disease is often considered to be incurable. It is important for people with this type of cancer to work closely with their healthcare team to determine the best course of treatment and to manage their symptoms.”	“Pembrolizumab is a type of cancer treatment called an immunotherapy. It can cause a number of side effects, including fever. If you are experiencing fever after receiving pembrolizumab, you should contact your doctor. They will be able to determine whether you need to go to the hospital based on the severity of your symptoms and your overall health. It is always better to be safe and get checked out by a doctor if you are experiencing any unusual symptoms.”
ChatGPT response 5	“Coughing can sometimes be a sign of lung cancer, but it is not always the case. Coughing can be caused by many different things, including colds, bronchitis, and other respiratory infections. In some cases, a cough can be a sign of something more serious, such as lung cancer. It is important to consult with a doctor if you have a persistent cough, especially if it is accompanied by other symptoms such as shortness of breath, chest pain, or blood in your phlegm. A doctor can determine the cause of your cough and recommend the appropriate treatment.”	“The prognosis of metastatic pancreatic cancer, also known as stage 4 pancreatic cancer, is generally not very good. This is because by the time pancreatic cancer has reached this advanced stage, it has typically spread to other parts of the body and is more difficult to treat. Most people with stage 4 pancreatic cancer survive for less than six months after being diagnosed, and only a small percentage of people live for more than one year. However, it is important to remember that every person is different, and some people may respond better to treatment than others. It is always best to discuss your specific situation with your doctor.”	“Pembrolizumab is a type of cancer treatment that is known to cause side effects in some people, including fever. If you are experiencing a fever after receiving pembrolizumab, you should contact your healthcare provider for advice on what to do. Depending on the severity of your symptoms, they may recommend that you go to the hospital. It is important to follow your healthcare provider’s instructions and to seek medical care if you are experiencing any severe or concerning symptoms.”

1. Is coughing a sign of lung cancer?

2. What is the prognosis of metastatic pancreatic cancer?

3. Does pembrolizumab cause fever, and should I go to the hospital?

For question 1, Google’s feature snippet indicated that cough, particularly a long-standing one (>3 weeks), is a main symptom of lung cancer—linking to the UK National Health Service website (Table 1). From ChatGPT, all 5 responses similarly indicated that cough is a lung cancer symptom, with 4 of 5 responses specifically referencing to a persistent or long-standing cough. Notably, all 5 ChatGPT responses expressed views that coughing is also a symptom of many other conditions and that referral to a doctor is required to get a proper diagnosis.

For question 2, Google’s feature snippet indicated that metastatic pancreatic cancer has a 5-year survival rate of 1%, with an average survival of about 1 year—linking to the Johns Hopkins Medicine website (Table 1). The 5 ChatGPT responses expressed that metastatic pancreatic cancer is a serious and advanced disease to which prognosis is generally poor or not very good, with 4 of 5 responses referencing to a survival typically less than 6 months or only a few months (the fifth response indicated survival rate “is very low”). Notably, 4 ChatGPT responses expressed nuances that individual prognoses can be different and that patients should consult with their health-care team to understand their disease. One ChatGPT response didn’t include a statement to discuss the topic with a health-care professional; this was also true of Google’s feature snippet.

For question 3, Google did not provide a feature snippet, relying on the user to find information. The first link was to pembrolizumab’s product webpage (Table 1), which stated that you should call your health-care provider right away if you develop any signs or symptoms noted on the serious side-effect list, which includes fever. To question 3, ChatGPT demonstrated variability in quality and content across the 5 recorded responses. In all 5 responses ChatGPT did recommend speaking to a health-care professional, the suggested time frames ranged from “good idea,” “you should contact,” “strongly recommended,” and do so “immediately.” Further, in 1 response, ChatGPT indicated that it did not have access to specific information about pembrolizumab effects, and in another, it indicated fever was not a common side effect, failing to explicitly state that it can occur. These contrast with 3 responses indicating fever is a side effect of pembrolizumab, yet in these 3 responses, ChatGPT failed to explicitly state the potential severity.

Overall, ChatGPT demonstrated an ability to formulate interpretable responses, which were similar in quality and content to Google’s feature snippet. Arguably, the additional context often provided by ChatGPT appeared to minimize the likelihood of alarm while providing action items. Nonetheless, for the question that Google did not produce a snippet, ChatGPT performed poorly, which may in part be resultant of the question wording.

Key considerations to be aware of when using ChatGPT are that 1) it does not provide references (or if it does they are not always true), 2) there is no single answer to a question (eg, answer will differ with the same prompt, as well as the preceding conversation and/or context), 3) ChatGPT is not connected to the internet (currently trained to Q4 2021), and 4) ChatGPT can produce incorrect answers in a confident sounding manner (for example, made up references) (1,5). The latter is an important required improvement to ensure the virtual assistant can respond with uncertainty when it is uncertain.

The rapid advancement and extensive interest in AI chatbots signal that we will see a proliferation of increasingly capable virtual assistants, including specialized health versions (2,3). The ChatGPT use case presents an urgent need for regulators and health-care professionals to be involved in developing standards for minimum quality and to raise patients’ awareness of current limitations (6).

Data availability

All data is available in the manuscript. Presented data, information, and responses from both chatGPT and Google were generated in the week of the 12th of December, 2022.

Author contributions

Ashley M Hopkins, PhD (Conceptualization; Data curation; Formal analysis; Funding acquisition; Investigation; Methodology; Project administration; Resources; Software; Supervision; Validation; Visualization; Writing—original draft; Writing—review & editing); Jessica M Logan, PhD (Conceptualization; Formal analysis; Funding acquisition; Investigation; Methodology; Supervision; Writing—original draft; Writing—review & editing); Ganessan Kichenadasse, MBBS, PhD (Formal analysis; Funding acquisition; Investigation; Methodology; Project administration; Supervision; Writing—original draft; Writing—review & editing); and Michael J. Sorich, PhD (Conceptualization; Data curation; Formal analysis; Funding acquisition; Investigation; Methodology; Project administration; Resources; Supervision; Writing—original draft; Writing—review & editing).

Funding

AMH is supported by an Emerging Leader Investigator Grant from the National Health and Medical Research Council, Australia (APP2008119). MJS is supported by a Beat Cancer Research Fellowship from Cancer Council South Australia.

Conflicts of interest

MJS report grants from Pfizer, outside the submitted work. The other authors have no relevant conflicts of interest to disclose for this paper.