Analysis of ChatGPT’s Responses to Patient Questions on Hidradenitis Suppurativa

Dear Editor:

Since ChatGPT’s release in November 2022, the large language chatbot has been used to diagnose medical conditions, write research papers, and assess clinical skills.¹ Hidradenitis suppurativa (HS) is a medical condition characterized by painful skin abscesses and scarring of the skin.² Because of the associated pain, sensitive anatomic locations, drainage, odor, and scarring, HS has a powerful psychosocial impact and decreases quality of life for patients.²

With the rise of artificial intelligence, the goal of this study was to analyze ChatGPT’s responses to patient questions on HS, particularly those dealing with psychosocial impact. Readability was assessed by number of sentences, words per sentence, characters per sentence, percentage of difficult words, Flesch Reading Ease (FRE), and the Flesch-Kincaid Grade Level (FKGL). The FRE score ranges from 0 (unreadable) to 100 (extremely easy to read).³ The FKGL provides a score corresponding to a United States education grade level, a lower score means a text is easier to read.³ For example, a text with a FKGL of 8.5 means that individuals with an education level of 8th grade or higher should be able to comprehend the text fully.³ This is the level that is suggested for health-related information to be considered accessible to all levels of literacy.³

ChatGPT’s responses were compared to responses to these questions available from the Hidradenitis Suppurativa Foundation website.⁴ Statistical significance was determined by a p-value <0.05 and 95% confidence intervals.

In January 2023, various questions were used to prompt ChatGPT, such as:

• “Will people think I am gross if I have hidradenitis suppurativa?”

• “In what ways can hidradenitis suppurativa impact my personal or work life?”

• “What can I do at home to help treat or prevent hidradenitis suppurativa flares?”

• “What should I expect over time if I have hidradenitis suppurativa?”

• “How do I explain hidradenitis suppurativa to my friends and family?”

• “Will hidradenitis suppurativa affect my romantic life?”

There was a statistically significant difference in the responses provided by ChatGPT and the HS Foundation in the words per sentence (p-value=0.0153, 95 CI: 1.58–10.05), percentage of difficult words (p-value=0.0002, 95% CI: 9.15–17.22), FRE (p-value=0.0002, 95% CI: -51.71– -27.27), and the FKGL (p-value=0.0004, 95% CI: 4.55–9.34).

The responses provided by the HS foundation to popular patient questions on HS are statistically significantly more accessible and easier to read than the responses provided by ChatGPT as determined by the percentage of difficult words, the Flesch Reading Ease, and The Flesch-Kincaid Grade Level. ChatGPT may still nevertheless be a source for patient information, however the responses provided by the HS foundation are more accessible and may reach a larger patient population. Limitations of this study include restrictions of the FRE and FKGL, these scoring methods do not take consider document factors such as, layout, pictures, chart, color, font, spacing, personal factors including motivation, information needs, or anxiety level, and style of writing such as cultural sensitivity.³ Other limitations include a relatively small size of questions asked and differing AI responses to the same prompt.