Artificial Intelligence-Based ChatGPT Responses for Patient Questions on Optic Disc Drusen

Abstract

Introduction

Optic disc drusen (ODD) are acellular deposits in the optic nerve head, which are most often benign and asymptomatic. Patients may develop visual field defects and be at increased risk of ischemic co-morbidities. As ODD can be difficult to distinguish from papilledema, patients are at risk of unnecessary clinical workups. Patient information is a key aspect of ODD management. In this study, we explored the accuracy of ChatGPT responses for typical patient questions on ODD.

Methods

Two content experts reached consensus on 20 typical patient questions. We retrieved five separate responses for each question from ChatGPT, totaling 100 responses. Each response was evaluated on a 5-point Likert-scale on accuracy by each content expert in an individual fashion.

Results

The two experts were in fair/substantial agreement in the evaluation of responses (Cronbach’s alpha: 0.64). Of the 100 responses, 17 were relevant and without any inaccuracies, 78 were relevant and with inaccuracies not being harmful, and five were relevant and with inaccuracies potentially harmful. The lowest accuracy scores were obtained for questions dealing with treatment and prognosis.

Conclusions

ChatGPT often provides relevant answers for patient questions on ODD, but inaccuracies become potentially harmful when questions deal with treatment and prognosis.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40123-023-00800-2.

Key Summary Points

Why carry out this study?

Optic disc drusen are acellular deposits in the optic nerve head, which affect approximately 2% of the population.

Currently, no evidence-based treatments are available to prevent or ameliorate optic disc drusen-related vision loss. The important role of the ophthalmologist here is to provide information on various aspects of this condition.

We explored the accuracy of ChatGPT responses for typical patient questions on optic disc drusen.

What was learned from the study?

ChatGPT often provides relevant answers for typical patient questions on optic disc drusen. On topics related to therapy, ChatGPT provides potentially harmful advice.

It is important for clinicians to understand the quality and accuracy of the information to which patients are exposed to better understand patient requests and patient behavior.

Introduction

Optic disc drusen (ODD) are acellular deposits in the optic nerve head, which are often benign and asymptomatic, and affect approximately 2% of humans [1–3]. Some patients develop visual field defects, and ODD increase the risk of vascular co-morbidities [4–9]. ODD are not necessarily visible on fundoscopy, and in these cases ODD can mimic a potentially life-threatening papilledema, leading to an unnecessary range of clinical workups. The diagnosis of ODD is best established using enhanced depth imaging optical coherence tomography [10].

There are currently no evidence-based treatments available to prevent or ameliorate ODD-related visual loss. Therefore, when the patients receive a neuroophthalmological examination and obtain a diagnosis of ODD, one of the most important roles for the clinician is to inform the patient on the various aspects of the disease and answer the patient’s questions.

Seeking information on health and disease on the Internet is a natural part of coping and living with a disease [11]. Interestingly, one Danish study reported that when information on the Internet was not in alignment with the information from the physician, patients reported dissatisfaction with consultations [12]. This means that as a physician, one needs to understand what information the patients are exposed to in order to understand the context from where the patient’s questions arise. Unfortunately, information on the Internet is rarely developed in collaboration with clinical experts, which put patients at risk of not only inaccurate information but also potentially dangerous information [13, 14].

ChatGPT (OpenAI, San Francisco, CA, USA), which stands for Chat Generative Pre-trained Transformer, was launched in December 2022 and is an artificial intelligence-based chatbot [15]. User statistics revealed a record fast adaptation rate across the world, and it is currently being used for a variety of applications [15]. In ODD, considering that no treatments are currently available, and access to a neuro-ophthalmologist is rarely readily available, one potentially very useful application would be if it could provide reasonable answers to common patient questions. Considering the current and increasing adaptation rate of ChatGPT, it becomes increasingly important to understand the accuracy of the answers provided by ChatGPT.

The objective of this study was to evaluate the accuracy of ChatGPT responses to common patient questions regarding ODD. This was made by reaching consensus on the most common questions from two ODD experts, by obtaining ChatGPT responses to these questions, and finally by evaluating the accuracy of these responses.

Methods

This study was designed as a study of publicly accessible software (ChatGPT) and we neither obtained nor analyzed clinical records or samples from patients or other study subjects. According to Danish law, such studies do not require institutional review board approval. All aspects of this study adhered to the tenets of the Declaration of Helsinki. According to ChatGPT Terms of Use, studies that do not deal with security flaws or inappropriate content can be conducted without a priori or a posteriori approval [16].

Question Development

The author group, including two authors (L.M. and S.H.) who are Optic Disc Drusen Studies consortium members (both with years of experience in ODD research and management) [17], developed the questions. All questions were sent to all authors and discussed until consensus to obtain face validity. We neither had a minimum nor maximum number of questions for this study, but discussed potential questions in the author group until consensus could be reached regarding relevance and importance of the questions. The following 20 questions were included for the analyses:

1. What is optic disc drusen?

2. How is optic disc drusen diagnosed?

3. What is the best way for diagnosing optic disc drusen?

4. Why do I have optic disc drusen?

5. Are my optic disc drusen inherited from my parents?

6. How are optic disc drusen formed?

7. Do optic disc drusen enlarge when I get older?

8. Is optic disc drusen preventable?

9. Is optic disc drusen treatable?

10. Can optic disc drusen impact my vision?

11. Will my optic disc drusen prevent me from driving a car?

12. Will I get blind from optic disc drusen?

13. Is there anything I should avoid when I have optic disc drusen?

14. Is there anything I can do to avoid the optic disc drusen from getting worse?

15. My doctor tells me I have pseudo-papilledema due to optic disc drusen. How can my doctor be sure that I don't have true papilledema in addition to my optic disc drusen?

16. Do I need regular eye examinations for my optic disc drusen?

17. I have a relative with optic disc drusen. Do I need to get an eye exam?

18. Should I be worried that my children will inherit optic disc drusen?

19. What are the symptoms of optic disc drusen?

20. Can optic disc drusen lead to other diseases?

ChatGPT Responses, Evaluation, and Data Analysis

We used ChatGPT version 4, which was published in March 2023 [18]. We accessed the Application Programming Interface using a custom Python (v. 3.8.13) script to ask the 20 questions, separately and without the answers to certain questions being able to influence the answers to other questions. This was ensured by starting a new conversation every time a new question was asked. Each question was asked five times separately to allow evaluation of the spectrum of different answers for the same question. In total, 100 answers were obtained for evaluation. The accuracy of obtained answers was evaluated on a 5-point Likert scale by two Optic Disc Drusen Studies consortium members (L.M. and S.H.). We defined individual points of the Likert scale for a consistent evaluation:

1. Irrelevant response/no response

2. Relevant response with major inaccuracies and potential for harm

3. Relevant response with major inaccuracies and no potential for harm

4. Relevant response with minor inaccuracies and no potential for harm

5. Relevant response without any inaccuracies

The two experts provided the evaluations in an independent fashion. We used Cronbach’s alpha to evaluate the inter-rater agreement between the experts. Responses were summarized using descriptive statistics to understand the distribution of responses. The scores for the five responses for each question from each expert was summarized using median and range. Statistics were made using SPSS version 28.0.1.0 (IBM, Armonk, NY, USA).

Results

All ChatGPT responses are summarized in the Supplementary Materials. The two experts exhibited a fair/substantial agreement in the evaluation of responses (Cronbach’s alpha: 0.64; 95% confidence interval 0.46–0.76). A comparison of the distribution of scores between the two experts is provided in Table 1.

Table 1.

Comparison of the distribution of scores between the two experts for all ChatGPT responses collected (n = 100)

Expert 2	Expert 1
	1 point	2 points	3 points	4 points	5 points
1 point	0	0	0	0	0
2 points	0	0	0	0	0
3 points	0	5	18	14	9
4 points	0	0	3	6	15
5 points	0	0	1	12	17

Scores were defined as follows: 1 point was given for “Irrelevant response/no response”, 2 points were given for “Relevant response with major inaccuracies and potential for harm”, 3 points were given for “Relevant response with major inaccuracies and no potential for harm”, 4 points were given for “Relevant response with minor inaccuracies and no potential for harm”, and 5 points were given for “Relevant response without any inaccuracies”

We experienced no absent responses or responses that were completely irrelevant. All responses were relevant but had varying degrees of inaccuracies.

Seventeen responses were rated as being without any inaccuracies by both experts. Below is one example of such a response to the question “My doctor tells me I have pseudo-papilledema due to optic disc drusen. How can my doctor be sure that I don't have true papilledema in addition to my optic disc drusen?”:

“To determine if someone with optic disc drusen also has true papilledema, an ophthalmologist would look for signs such as increased intracranial pressure or other symptoms of a neurological condition. Imaging tests such as an MRI or CT may be done to rule out any underlying brain issues that could be causing the papilledema. Additionally, they may perform regular follow-up visits and evaluations to monitor any changes in your symptoms and condition over time. It is important to consult with your doctor and get proper diagnosis and treatment for your condition.”

In 33 other responses, one or both experts stated the presence of minor inaccuracies without potential for harm. One example of such minor inaccuracy is illustrated in the following response to the question “How are optic disc drusen formed?”:

“Optic disc drusen are formed due to the aggregation of calcium and protein deposits in the optic nerve head (ONH). The exact cause behind this process is not known, but it is believed that the accumulation of debris may be related to genetic factors or problems with the normal processes of protein metabolism and drainage within the eye. As these deposits build up over time, they can result in optic nerve abnormalities and potential vision problems. Although there is currently no cure for optic disc drusen, early detection and appropriate monitoring of patients can help reduce the risk of vision loss.”

This response suggested that early detection and monitoring could alter the natural course of the disease, which was deemed inaccurate. The experts deemed this as being a minor inaccuracy and without any potential for harm. Similarly, 50 responses, or half of all responses were either without any inaccuracies or with only minor inaccuracies.

In 45 responses, one or both experts stated the presence of major inaccuracies without any potential for harm. One example of major inaccuracy is illustrated in the following response to the question “Will I get blind from optic disc drusen?”:

“Optic disc drusen typically do not cause blindness. However, they can cause visual field defects or symptoms such as blurry vision or blind spots that may interfere with daily activities. In rare cases, optic disc drusen can lead to complications such as papilledema or ischemic optic neuropathy, which can lead to vision loss. It is important to discuss any concerns about your eye health with a trusted healthcare professional.”

This response linked papilledema as a rare complication of ODD, which was deemed incorrect. The experts evaluated this to be a major inaccuracy, but without any potential for harm.

Finally, five responses were evaluated as having major inaccuracies by both experts, and one expert stated potential for harm. These responses are all provided with a detailed explanation of the potential for harm in Table 2. Thus, five responses, or 1 in 20, had any potential for harm.

Table 2.

Responses deemed as having major inaccuracies with any potential for harm

Question	Response	Reason for harm
What is optic disc drusen?	“Optic disc drusen is a condition in which small, whitish-yellow deposits of calcium or other minerals accumulate within the optic nerve head. This can cause varying degrees of vision loss or visual field defects, as well as other symptoms like headaches or eye pain.”	This response indicates that headache and eye pain may be caused by ODD. This is not only inaccurate but may also delay a diagnosis of other reasons for headache and eye pain, which may even require timely treatment, and is therefore a response with potential for harm
Is optic disc drusen treatable?	“Unfortunately, there is no known cure for optic disc drusen. However, the symptoms can be managed with regular eye checkups, monitoring of vision changes, and treating associated eye conditions such as glaucoma or anterior ischemic optic neuropathy if they arise. In rare cases where optic disc drusen cause severe vision problems, surgical options may be explored, although the success rate of these procedures is not well established. It is important to speak with an ophthalmologist for individualized recommendations and management of optic disc drusen.”	This response indicates that surgery may be a treatment option for ODD. This is inaccurate and may lead to patients seeking and exploring surgical options from dubious healthcare providers and is therefore a response with potential for harm
Is optic disc drusen treatable?	“There is no known cure for optic disc drusen, but treatment is usually focused on monitoring the condition and managing any associated symptoms. Your doctor may recommend regular eye exams to track any changes in your vision, as well as imaging tests like optical coherence tomography (OCT) or visual field testing. In some cases, treatment may be recommended to alleviate symptoms such as visual disturbances or headaches. This may include medications or eyeglasses to help with focusing or reducing glare, or other vision aids. In rare cases where optic disc drusen are causing significant vision loss, surgery may be considered, although this is not often recommended. It's important to follow your doctor's advice and attend all recommended check-ups to ensure that any changes to your vision can be detected and managed appropriately.”	This response indicates that surgery may be a treatment option for ODD. This is inaccurate and may lead to patients seeking and exploring surgical options from dubious healthcare providers and is therefore a response with potential for harm
Can optic disc drusen impact my vision?	“Yes, optic disc drusen can affect your vision. Optic disc drusen are deposits that accumulate in the optic nerve head of a person's eye. These drusen can significantly impact visual function and lead to a range of symptoms, including peripheral visual field loss, blurred or distorted vision, double vision, or visual anomalies such as blind spots or scotomas. Although optic disc drusen are not associated with any specific eye disease, they can increase the risk of developing glaucoma, a condition that can cause permanent damage to the optic nerve if left untreated. Therefore, if you experience any vision-related symptoms, it is essential to consult an eye doctor for proper diagnosis and treatment.”	This response states that optic disc drusen can increase the risk of developing glaucoma. This is not only inaccurate but may also lead to life-long, expensive, and unnecessary anti-glaucoma treatment, including surgery, and is therefore a response with potential for harm
Can optic disc drusen lead to other diseases?	“Optic disc drusen (ODD) are small, hyaline deposits of calcium found within the optic nerve head. While ODD itself is not considered a disease, having it may increase the risk of developing certain eye conditions. These potential conditions include: 1. Anterior Ischemic Optic Neuropathy (AION): AION occurs when blood flow to the optic nerve is compromised, causing optic nerve damage and vision loss. People with ODD may be at an increased risk of developing AION 2. Non-arteritic anterior ischemic optic neuropathy (NAION): NAION is a sudden loss of vision in one eye that can occur without warning. It may be associated with underlying vascular diseases like diabetes, hypertension, high cholesterol or pre-existing optic nerve abnormalities 3. Glaucoma: ODD can mimic glaucomatous optic neuropathy by causing cupping of the optic nerve head which can lead to excessive eye pressure 4. Pseudo-optic atrophy: ODD can also lead to optic disc swelling and shrinkage which may resemble optic atrophy While these risks are present, they are not very common. People who have ODD will need regular eye exams to monitor their eye health over time. If you have any concerns about your eye health, talk to an eye care professional to help understand your risks and how best to manage them.”	This response lists both anterior ischemic optic neuropathy (AION) and non-arteritic anterior ischemic optic neuropathy (NAION), which gives the impression that these are two different disease entities. NAION is a subgroup of AION that is non-arteritic. Another subgroup of AION is arteritic AION, often abbreviated as AAION, which is a different disease entity related to arteritic disease such as giant cell arteritic. Stating AION and NAION as two separate diseases may confuse and lead to an understanding of AAION being related to ODD, which may delay important time of diagnosis and treatment for AAION and is therefore a response with potential for harm. Also, the response states that ODD can cause cupping of the optic nerve head mimicking glaucomatous optic neuropathy. This is not only wrong, but may also lead to life-long, expensive, and unnecessary anti-glaucoma treatment, including surgery, and is therefore a response with potential for harm. Finally, the response states that ODD can lead to pseudo-optic atrophy, since they can both cause optic disc swelling and shrinkage. The term pseudo-atrophy is not an established term among physicians, and it is inaccurate and confusing. Also, optic nerve shrinkage is an imprecise terminology potentially also leading to unnecessary auxiliary workup and a potential for harm

Accuracy of the responses for each question are summarized in Table 3. The combined scores ranged from a mean of 4.8 ± 0.4 for the question with the highest scores to a mean of 3.1 ± 0.3 for the question with the lowest scores. The highest scores were obtained for the questions “Do I need regular eye examinations for my optic disc drusen?”, “Will my optic disc drusen prevent me from driving a car?”, “Is optic disc drusen preventable?”, “My doctor tells me I have pseudo-papilledema due to optic disc drusen. How can my doctor be sure, that I don't have true papilledema in addition to my optic disc drusen?”, and “How is optic disc drusen diagnosed?”. The lowest scores were obtained for the questions “Is there anything I can do to avoid the optic disc drusen from getting worse?”, “Can optic disc drusen impact my vision?”, “What are the symptoms of optic disc drusen?”, “Can optic disc drusen lead to other diseases?”, and “Is optic disc drusen treatable?”.

Table 3.

Accuracy scores for ChatGPT responses for each of the 20 questions

Question	Expert 1 scores	Expert 2 scores	Combined scores
1. What is optic disc drusen?	5 (2–5)	4 (3–5)	3.9 (0.6)
2. How is optic disc drusen diagnosed?	5 (4–5)	4 (3–5)	4.4 (0.7)
3. What is the best way for diagnosing optic disc drusen?	4 (4–4)	4 (3–5)	3.9 (0.6)
4. Why do I have optic disc drusen?	4 (3–5)	3 (3–5)	3.9 (0.9)
5. Are my optic disc drusen inherited from my parents?	4 (4–4)	5 (3–5)	4.3 (0.7)
6. How are optic disc drusen formed?	4 (3–5)	3 (3–4)	3.5 (0.7)
7. Do optic disc drusen enlarge when I get older?	4 (4–5)	5 (3–5)	4.4 (0.7)
8. Is optic disc drusen preventable?	5 (5–5)	5 (3–5)	4.6 (0.8)
9. Is optic disc drusen treatable?	3 (2–5)	3 (3–4)	3.4 (1.1)
10. Can optic disc drusen impact my vision?	3 (2–5)	3 (3–3)	3.2 (0.8)
11. Will my optic disc drusen prevent me from driving a car?	5 (5–5)	4 (4–5)	4.7 (0.5)
12. Will I get blind from optic disc drusen?	3 (3–5)	4 (3–5)	3.9 (1.0)
13. Is there anything I should avoid when I have optic disc drusen?	5 (4–5)	3 (3–5)	4.0 (0.9)
14. Is there anything I can do to avoid the optic disc drusen from getting worse?	3 (3–3)	3 (3–4)	3.1 (0.3)
15. My doctor tells me I have pseudo-papilledema due to optic disc drusen. How can my doctor be sure that I don't have true papilledema in addition to my optic disc drusen?	5 (4–5)	4 (3–5)	4.5 (0.7)
16. Do I need regular eye examinations for my optic disc drusen?	5 (4–5)	5 (4–5)	4.8 (0.4)
17. I have a relative with optic disc drusen. Do I need to get an eye exam?	4 (4–5)	4 (3–5)	4.1 (0.6)
18. Should I be worried that my children will inherit optic disc drusen?	3 (3–5)	5 (3–5)	4.0 (0.9)
19. What are the symptoms of optic disc drusen?	3 (3–4)	3 (3–3)	3.2 (0.4)
20. Can optic disc drusen lead to other diseases?	3 (2–5)	3 (3–3)	3.2 (0.8)

Scores for each of the five responses are summarized for each question. Individual expert scores are reported in median and range. Combined scores are reported in mean and standard deviation

Discussion

In this study, we evaluated the accuracy of ChatGPT responses for typical patient questions regarding ODD. Overall, ChatGPT responded relevant to all questions. The accuracy of the answers was high for a substantial part, and major inaccuracies with potential for harm was only an issue for a small minority of the responses. Nevertheless, it is important for clinicians to understand the nature of such inaccuracies.

In previous studies, we explored the accuracy of ChatGPT responses for questions dealing with common retinal diseases and for vernal keratoconjunctivitis [19, 20]. For retinal diseases, we experienced inaccuracies and the potential for harm to be related to questions dealing with treatment [19]. Similarly, treatment-related inaccuracies were observed for responses related to vernal keratoconjunctivitis [20]. However, in both studies, ChatGPT was able to provide accurate information on disease definition and diagnosis [19, 20]. These patterns, which we also observed in this study, highlight the key issues in obtaining health information from an artificial intelligence-based model that is developed as a large language model [22, 23]. Large language models are models that consist of a neural network that is trained on very large datasets of text [22, 23]. These models allow for accurate response and restructure of text that exists in relatively large quantities, e.g., textbooks, general information on the web, Wikipedia, etc.; but become challenged in providing responses that deal with details related to treatment and monitoring [22, 23]. Treatment and monitoring are often aspects that undergo changes faster than that of disease definition and pathophysiology, and clinical knowledge is often based on current general expert consensus or guidelines. These aspects are difficult to incorporate in an artificial intelligence-based large language model, as one important limitation of these generative artificial intelligence-based systems is that they rely on pre-existing data to learn and then use to synthesize new output.

The literature on health information-seeking behaviors outline a complex field with many influencing factors [11, 23]. Studies also find changes in trends with time [11, 23], which to some extent may be explained by increasing technology adaptation with time. Thus, health information-seeking behavior is likely to be a dynamic phenomenon over time, and chatbot-based information may represent a new topic within this field. Advantages of current artificial intelligence-based large language models include prompt engineering (i.e., providing a setting prior to asking the questions, defining the output-style of the responses, or a dialogue-based conversation), which may allow for a more precise input and a more relevant output, at least from the patients’ perspective.

Strengths and limitations need to be acknowledged when interpreting the results of our study. Questions were developed as a consensus and face validity approach by two content experts. The two content experts being affiliated with the same ophthalmological department constitutes a limitation, as they theoretically would be more likely to be in agreement than two experts not affiliated with the same department. The approach of developing questions based on expert input is feasible but may not always incorporate the questions that the patients may prioritize. Thus, there is also a limitation in that patients were not involved in question development. Another important limitation is that responses of ChatGPT vary. To better understand the variety of responses, we retrieved five responses for each question, which in our experience allows for a certain saturation in the quality of responses. Finally, one source of inaccuracy is that differences between the experts’ interpretation of text may lead to different responses on the Likert scale, which is also seen in our study, as the experts were not in full agreement.

Conclusions

In conclusion, this study explored the accuracy of ChatGPT responses for typical patient questions on ODD and found that ChatGPT overall provides relevant answers. In 1 of 20 responses, we detected inaccuracies with potential for harm. Considering that the neuro-ophthalmologist, or even the ophthalmologist, not being readily available, but ChatGPT being able to respond within seconds, the patients will likely obtain information from ChatGPT or similar services. As clinicians, what we can do is to understand the nature of the information provided to the patients in our effort to guide and advise patients most appropriately.

Supplementary Information

Below is the link to the electronic supplementary material.

Author Contributions

Conceptualization, Ivan Potapenko, Lasse Malmqvist, Yousif Subhi, and Steffen Hamann; Methodology, Ivan Potapenko and Yousif Subhi; Formal analysis and investigation, Ivan Potapenko, Lasse Malmqvist, Yousif Subhi, and Steffen Hamann; Writing—original draft preparation, Ivan Potapenko, Lasse Malmqvist, Yousif Subhi, and Steffen Hamann; Writing—review and editing, Ivan Potapenko, Lasse Malmqvist, Yousif Subhi, and Steffen Hamann; Supervision, Steffen Hamann. All authors have read and approved the final manuscript.

Funding

No funding or sponsorship was received for this study or publication of this article.

Data Availability

All data generated or analyzed during this study are included as supplementary information files.

Declarations

Conflict of Interest

Yousif Subhi has received speakers fees from Bayer and Roche, not related to this work. Ivan Potapenko, Lasse Malmqvist, and Steffen Hamann have nothing to disclose.

Ethical Approval

This study was designed as a study of publicly accessible software (ChatGPT) and we neither obtained nor analyzed clinical records or samples from patients or other study subjects. According to Danish law, such studies do not require institutional review board approval. All aspects of this study adhered to the tenets of the Declaration of Helsinki.