Reply to the Letter to the Editor: Is Information About Musculoskeletal Malignancies From Large Language Models or Web Resources at a Suitable Reading Level for Patients?

To the Editor,

We thank Drs. Kleebayoon and Wiwanitkit for their comments in their letter to the editor [3] regarding our article, “Is Information About Musculoskeletal Malignancies From Large Language Models or Web Resources at a Suitable Reading Level for Patients?” [2].

The authors first mention the potentially limited scope of the study design because we used queries from only one source, the American Cancer Society (ACS), to compare both ChatGPT and Google. We selected these queries because they were purposely broad in nature and mimicked the most fundamental questions a patient may ask regarding this subject matter. Recommendations from Kleebayoon and Wiwanitkit [3] to incorporate more data sources in addition to patient-generated questions are helpful suggestions for future studies on this topic.

Kleebayoon and Wiwanitkit [3] also mention the lack of patient input into ChatGPT and Google responses to assess the real-world perceptions of the collected data. This is another beneficial suggestion for future research that builds upon our study’s findings. As for our study [2], we felt it was necessary to conduct an initial study that quantitatively assessed readability metrics of these online tools and compare the outcomes to other studies that investigated readability of web-based orthopaedic educational materials. Due to our study questions and methodology, this investigation did not include the use of human participants, and was thus exempt from IRB approval.

Kleebayoon and Wiwanitkit [3] suggested future investigations include “other presenting styles,” such as summaries or graphs, to gauge what the most-effective method of display information is. While this is generally outside the scope of our study [2], we did discuss various recommendations made by the NIH to improve the readability and understanding of information intended for patients, including the use of one- to two-syllable words and avoiding complex graphs and tables. We also dedicated a section to discussing Google features such as “Knowledge Graph” and “Featured Snippet,” which utilize Google’s algorithm to provide users with summarized information from different websites upon inputting a search query. But these features, while potentially helpful tools, could not be used in our study for the following reasons:

1. ChatGPT does not currently generate graphical summaries based on queries.

2. Google features do not generate for every search query.

Kleebayoon and Wiwanitkit [3] also recommend including other AI models as a comparison to ChatGPT-3.5 to assess future improvements in AI-generated patient education materials. At the time our study was conducted, ChatGPT was the most widely used generative AI system [4], deeming it the most appropriate initial comparator to Google search. The continued developments in the field of AI will allow future studies to investigate the authors’ suggestion in determining the potential growth made in AI use for medical education.

Lastly, Kleebayoon and Wiwanitkit [3] describe a future use of AI to adjust responses based upon the cognitive level of the user. Our study included a reference [1] that used response customization to improve readability for health-related queries made by patients. However, when applied to the average readability level seen within our study [2], this technology did not produce a response that met NIH guidelines, prompting us not to include user-customized queries. The initial investigations made into tailored AI-generated response should continue to be investigated as the AI models improve over time.