Quality of the Information provided by ChatGPT for Patients in Breast Plastic Surgery: Are we already in the future?

Abstract

Introduction

In recent years, artificial intelligence (AI) has gained popularity, even in the field of plastic surgery. It is increasingly common for patients to use the internet to gather information about plastic surgery, and AI-based chatbots, such as ChatGPT, could be employed to answer patients' questions.

The aim of this study was to evaluate the quality of medical information provided by ChatGPT regarding three of the most common procedures in breast plastic surgery: breast reconstruction, breast reduction, and augmentation mammaplasty.

Methods

The quality of information was evaluated through the expanded EQIP scale. Responses were collected from a pool made by ten resident doctors in plastic surgery and then processed by SPSS software ver. 28.0.

Results

The analysis of the contents provided by ChatGPT revealed sufficient quality of information across all selected topics, with a high bias in terms of distribution of the score between the different items. There was a critical lack in the “Information data field” (0/6 score in all the 3 investigations) but a very high overall evaluation concerning the “Structure data” (>7/11 in all the 3 investigations).

Conclusion

Currently, AI serves as a valuable tool for patients; however, engineers and developers must address certain critical issues. It is possible that models like ChatGPT will play an important role in improving patient's consciousness about medical procedures and surgical interventions in the future, but their role must be considered ancillary to that of surgeons.

Introduction

Artificial intelligence (AI) can be defined as the study of algorithms that provide machines with the capability to reason and execute cognitive functions, including problem-solving, object and word recognition, and decision-making.¹

In recent years, AI has gained popularity, extending its influence to the field of medicine and surgery. Different subcategories of AI include machine learning, deep learning, natural language processing, and facial recognition that could be applied in plastic surgery.^2,3

For patients, it is becoming more and more common to use internet to gather medical information.^4,5 This information may be sourced from unreliable channels, potentially influencing the patient during medical consultations and impacting the decision-making process related to treatment choices.⁵

In this regard, there are different forms of AI, such as ChatGPT, which is a generative language model developed by OpenAI. Designed to comprehend and generate text in natural language, ChatGPT facilitates advanced conversational interactions with computers. Trained on a diverse array of internet texts, it possesses the ability to answer questions, including those pertaining to medical topics. The aim of this study is to evaluate the quality of medical information offered by ChatGPT to patients regarding three prevalent breast plastic surgery procedures: breast reconstruction, augmentation0020mammaplasty, and breast reduction.

Materials and methods

Assuming that some patients may consider ChatGPT a source of knowledge, inquiries were made to obtain general information about three commonly performed surgical procedures in plastic surgery: breast reconstruction, augmentation mammaplasty, and breast reduction.

The expanded EQIP^6,7 scale was applied to evaluate the quality of the information offered by ChatGPT. This scale comprises 36 questions divided into three sections: Content data (items 1–18), Identification data (items 19–24), and Structure data (items 25–36) with YES or NO as a possible answer.

The “Content data” field of the scale assesses the medical problem and the treatment alternatives, considering aspects such as side effects and complications. The “Identification data” section is focused on the name of the entities that produced the documents, bibliography, and date of issue or revision. Lastly, the “Structure data” pertains to the use of everyday language, short sentences, and clear information, aiming to ascertain the comprehensibility of the information for patients.

Each question holds a value of one point, resulting in a total score ranging from 0 to 36 points. A score of 18 or higher is categorized as a high score, while a score below 18 is deemed a low score. Responses were gathered between November 6 and November 9, 2023 from a pool of 10 independent resident doctors in plastic surgery. Each doctor was presented with the questionnaire, and the data considered for this study represent an average of their results. The evaluation questionnaire data were collected and analyzed using the statistical program SPSS software version 28.0 (IBM Corporation; Armonk, New York).

Results

The analysis of the content provided by ChatGPT revealed sufficient quality of information across all the selected topics, although there was a notable bias in the distribution of scores among the different items. Specifically, a critical deficiency was identified in the “Identification data,” while a positive assessment was observed in terms of the “Structure data.” The first analyzed research (Table 1) was focused on the “Breast reconstruction” topic, and it showed a total mean score of 19/36. The second one (Table 2) evaluated the “Breast reduction” and the mean score obtained was also 19/36. The last investigation regarded the “Augmentation mammaplasty” (Table 3), and the mean score was 20/36. None of the answers provided by ChatGPT included information about the date of issue or revision or bibliography. However, in terms of “Structure data,” the analysis of language and sentences revealed a very good quality with logical order, use of everyday language, explanations of complex words, and clear information.

Table 1.

EQIP tool results applied to “Breast reconstruction” information provided by ChatGPT.

Question	Yes (%)	No (%)	Response
Content Data
1.Initial definition of which subjects will be covered	70	30	Yes
2.Coverage of the above defined subjects	60	40	Yes
3.Description of the medical problem	90	10	Yes
4.Definition of the purpose of the medical intervention	100	0	Yes
5.Description of the treatment alternatives (including no treatment)	90	10	Yes
6.Description of the sequence of the medical procedure	0	100	No
7.Description of the qualitative benefits	90	10	Yes
8.Description of the quantitative benefits	0	100	No
9.Description of the qualitative risk and side effects	90	10	Yes
10.Description of the quantitative risk and side effects	0	100	No
11.Addressing quality of life issues	70	30	Yes
12.Description of how potential complications will be dealt with	0	100	No
13.Description of precautions that the patient may take	80	20	Yes
14.Mention of the alert signs that the patient may detect	0	100	No
15.Addressing medical intervention cost and insurance issues	100	0	Yes
16.Specific contact details for hospital services	0	100	No
17.Specific details of other sources of reliable information/support	0	100	No
18.The document covers all relevant issues on the topic	0	100	No
Identification Data
19.Date of issue or revision	0	100	No
20.Logo of the issuing body	0	100	No
21.Name of the persons or entities that produced the document	0	100	No
22.Name of persons or entities that financed the document	0	100	No
23.Short bibliography of evidence-based data used in the document	0	100	No
24.The document states if and how patients were involved/consulted in its production	0	100	No
Structure Data
25.Use of everyday language, explains complex words or jargon	100	0	Yes
26.Use of generic names for all medications or products	90	10	Yes
27.Use of short sentences	100	0	Yes
28.The document personally addresses the reader	90	10	Yes
29.The tone is respectful	100	0	Yes
30.Information is clear	100	0	Yes
31.Information is balanced between risk and benefits	70	30	Yes
32.Information is presented in a logical order	100	0	Yes
33.The design and layout are satisfactory	70	30	Yes
34.Figures and graphs are clear and relevant	0	100	No
35.The document has a dedicated space for the reader's notes	0	100	No
36.The document includes a consent form, contrary to recommendations	0	100	No

Table 2.

EQIP tool results applied to “Breast reduction” information provided by ChatGPT.

Question	Yes (%)	No (%)	Response
Content Data
1.Initial definition of which subjects will be covered	100	0	Yes
2.Coverage of the above defined subjects	90	10	Yes
3.Description of the medical problem	100	0	Yes
4.Definition of the purpose of the medical intervention	100	0	Yes
5.Description of the treatment alternatives (including no treatment)	20	80	No
6.Description of the sequence of the medical procedure	90	10	Yes
7.Description of the qualitative benefits	90	10	Yes
8.Description of the quantitative benefits	0	100	No
9.Description of the qualitative risk and side effects	90	10	Yes
10.Description of the quantitative risk and side effects	0	100	No
11.Addressing quality of life issues	90	10	Yes
12.Description of how potential complications will be dealt with	60	40	Yes
13.Description of precautions that the patient may take	80	20	Yes
14.Mention of the alert signs that the patient may detect	10	90	No
15.Addressing medical intervention cost and insurance issues	0	100	No
16.Specific contact details for hospital services	0	100	No
17.Specific details of other sources of reliable information/support	0	100	No
18.The document covers all relevant issues on the topic	0	100	No
Identification Data
19.Date of issue or revision	0	100	No
20.Logo of the issuing body	0	100	No
21.Name of the persons or entities that produced the document	0	100	No
22.Name of persons or entities that financed the document	0	100	No
23.Short bibliography of evidence-based data used in the document	0	100	No
24.The document states if and how patients were involved/consulted in its production	0	100	No
Structure Data
25.Use of everyday language, explains complex words or jargon	100	0	Yes
26.Use of generic names for all medications or products	90	10	Yes
27.Use of short sentences	100	0	Yes
28.The document personally addresses the reader	100	0	Yes
29.The tone is respectful	100	0	Yes
30.Information is clear	90	10	Yes
31.Information is balanced between risk and benefits	90	10	Yes
32.Information is presented in a logical order	100	0	Yes
33.The design and layout are satisfactory	90	10	Yes
34.Figures and graphs are clear and relevant	0	100	No
35.The document has a dedicated space for the reader's notes	0	100	No
36.The document includes a consent form, contrary to recommendations	0	100	No

Table 3.

EQIP tool results applied to “Augmentation mammaplasty” information provided by ChatGPT.

Question	Yes (%)	No (%)	Response
Content Data
1.Initial definition of which subjects will be covered	100	0	Yes
2.Coverage of the above defined subjects	90	10	Yes
3.Description of the medical problem	100	0	Yes
4.Definition of the purpose of the medical intervention	90	10	Yes
5.Description of the treatment alternatives (including no treatment)	80	20	Yes
6.Description of the sequence of the medical procedure	100	0	Yes
7.Description of the qualitative benefits	90	10	Yes
8.Description of the quantitative benefits	0	100	No
9.Description of the qualitative risk and side effects	70	30	Yes
10.Description of the quantitative risk and side effects	0	100	No
11.Addressing quality of life issues	100	0	Yes
12.Description of how potential complications will be dealt with	90	10	Yes
13.Description of precautions that the patient may take	90	10	Yes
14.Mention of the alert signs that the patient may detect	10	90	No
15.Addressing medical intervention cost and insurance issues	0	100	No
16.Specific contact details for hospital services	0	100	No
17.Specific details of other sources of reliable information/support	0	100	No
18.The document covers all relevant issues on the topic	10	90	No
Identification Data
19.Date of issue or revision	0	100	No
20.Logo of the issuing body	0	100	No
21.Name of the persons or entities that produced the document	0	100	No
22.Name of persons or entities that financed the document	0	100	No
23.Short bibliography of evidence-based data used in the document	0	100	No
24.The document states if and how patients were involved/consulted in its production
Structure Data
25.Use of everyday language, explains complex words or jargon	100	0	Yes
26.Use of generic names for all medications or products	100	0	Yes
27.Use of short sentences	100	0	Yes
28.The document personally addresses the reader	90	10	Yes
29.The tone is respectful	100	0	Yes
30.Information is clear	100	0	Yes
31.Information is balanced between risk and benefits	80	20	Yes
32.Information is presented in a logical order	100	0	Yes
33.The design and layout are satisfactory	90	10	Yes
34.Figures and graphs are clear and relevant	0	100	No
35.The document has a dedicated space for the reader's notes	0	100	No
36.The document includes a consent form, contrary to recommendations	0	100	No

Discussion

Patients are increasingly using the internet as a crucial means of acquiring knowledge on various topics.^8,9 Websites and platforms providing medical information are constantly changing and developing, resulting in a growing need to select high quality, reliable sources. In this field, AI is growing in popularity, and tools like ChatGPT could provide valuable support by offering detailed and up-to-date information, enhancing the communication between the surgeon and the patient.

The primary endpoint of the study was to examine the quality of information supplied by ChatGPT, a generative language model based on AI developed by OpenAI. To ensure objectivity and standardization in the evaluation, the EQIP scale^6,7 was employed. This scale was utilized for assessing responses concerning inquiries about three prevalent surgeries in plastic surgery: breast reconstruction, breast reduction, and augmentation mammaplasty. For each topic, information was requested by asking simple questions in the same form a patient would do it.

Breast reconstruction stands out as a primary focus in plastic surgery, undergoing continuous evolution in both surgical techniques and materials. There is a noticeable trend toward an increase in total mastectomies, coupled with immediate or delayed breast reconstruction, a practice proven to have a positive impact on the quality of life for patients.¹⁰ Numerous studies indicate that women undergoing mastectomy frequently encounter challenges related to body image, resulting in a decrease in their quality-of-life scores.¹¹ In these delicate patients, correct information plays a primary role to familiarize them with the surgery they will face, including complications. G. Lanzano¹² pointed out how the available resources and time constraints in clinical consultation may limit the amount of information that can be conveyed to these patients, particularly with regard to explanations about the complex and deep personal process they are going through. In this field, AI could make a significant impact by delivering comprehensive information.

The results obtained by the application of EQIP tool are shown in Table 1. The total score was 19/36.

Breast reduction is a surgery frequently sought by young patients, many of whom may be planning to have children later in life. The surgeon's goal is to provide a procedure that minimizes scars and preserves as much function as possible. However, it is imperative to inform patients about the potential risks associated with the procedure and the precautions they must take. For instance, advising patients to cease smoking is crucial to reduce the risks of complications related to peripheral vascularization deficit.¹³ The results obtained by the application of EQIP tool are shown in Table 2. The total score was 19/36.

Augmentation mammoplasty is a procedure sought by millions of women for aesthetic reasons. The steadily growing availability of this type of surgery, coupled with the swift and sometimes incomplete communication through social media, tends to make patients underestimate the risks and complications associated with this procedure. This is particularly concerning as these risks can be less well-tolerated in patients who perceive themselves as healthy. The results obtained by the application of EQIP tool are shown in Table 3. The total score was 20/36.

Healthcare professionals and reconstructive surgeons are tasked with playing a crucial role in guiding patients undergoing such procedures, directing them toward reliable and comprehensive sources of information. AI is an important tool rapidly gaining traction in various fields, including the medical domain. The current study delves into the quality of information supplied by a ChatBot (ChatGPT), aiming to assess the potential role this tool may play in the doctor-patient relationship.

This tool has demonstrated its capability to break down social barriers through the use of inclusive and easily understandable language. Moreover, it possesses the ability to overcome language barriers by effectively reprocessing information in different languages. Xie et al¹⁴ recently examined ChatGPT's responses to a series of hypothetical questions designed to simulate a consultation about rhinoplasty. The study demonstrated the ChatBot's ability to offer coherent and easily understandable answers, underscoring, however, the significance of an individualized approach.

In this regard, Giovanni Buzzaccarini et al¹⁵ described how AI can personalize treatment plans by analyzing patient data, leading to satisfactory results.

However, AI provides information that is reworked from undisclosed sources and lacks any bibliography or references. This absence can result in the provided information being perceived as less impartial, leaving the reader with no opportunity to develop critical thinking. The lowest score obtained in all three investigations was assigned to the “Identification data” section, with a score of 0/6.

To the best of our knowledge, this is the first study to evaluate the quality of patient information obtained from ChatGPT using a validated tool, specifically the EQIP instrument.

This study has some limitations. The scoring was conducted by a group of plastic surgery residents possessing a high level of knowledge but less experience in managing complications or addressing patient dissatisfaction compared to a consultant. It is plausible that an evaluation by an experienced plastic surgeon could reveal deficiencies in specific information, particularly in the “Content data” section. Additionally, the assessment was conducted based on the ChatBot's responses to questions in Italian, and it cannot be excluded that the quality of information may vary when provided in other languages. Finally, AI is inherently a trainable and improvable system, adapting based on the challenges it faces. It is conceivable that the quality and completeness of information provided by ChatGPT may improve in a short time, especially in the “Content data” field. Hence, we should view our study as a momentary snapshot of the information offered by ChatGPT.

Conclusion

AI has proven to be a crucial tool across various applications, including the medical field. Online healthcare information serves as a primary knowledge source for patients, and it is likely that models like ChatGPT will play a significant role in enhancing patient awareness about medical procedures and surgical interventions in the near future. This study provides an overview of the quality of information provided by ChatGPT, employing an objective evaluation through the EQIP scale. Generally, the information quality was deemed sufficient, with excellent scores regarding the form and comprehensibility of the data. However, critical issues related to the sources of information reported need to be addressed by engineers and developers in the future. Currently, AI represents an important tool, with its role considered ancillary to that of the surgeon, yet it can contribute significantly to improving the patient's journey through a surgical path.

Ethical approval

Not required.

Funding

No funding was provided for this work.

Declaration of competing interest

The authors report no conflict of interest.