ChatGPT as a source of information on asthma

To the Editor:

We hereby comment on the article by Høj et al titled, “Evaluating the scientific reliability of ChatGPT as a source of information on asthma.”¹ The study's goal of evaluating the scientific trustworthiness of ChatGPT as a source of asthma information is topical and important, considering the growing dependence on artificial intelligence (AI) for health-related inquiries. Although a large number of responses with a score of 4 or higher shows excellent accuracy, it is critical to eliminate any biases in assessment and question design. For example, questions based on the most common patient issues or addressing specific themes may distort perceived accuracy. Furthermore, how were the physicians chosen, and what biases may they have in evaluating AI-generated content? Investigating these aspects may help to improve the validity of the findings.

Although this approach is basic, it raises concerns about the breadth and depth of the examination. The use of an accuracy scale of 1 to 5 may oversimplify complex medical data, and a lack of specific assessment criteria for raters may result in subjective interpretations. It would be helpful if Høj et al¹ explained how they guaranteed that raters were consistent and objective. Furthermore, follow-up questions were recorded but not examined, resulting in major gaps in the depth of comprehension and usefulness of ChatGPT material. Future replication studies may benefit from a mixed-methods strategy that includes qualitative data to supplement quantitative conclusions.

In terms of novelty, studies can investigate the consequences of discovered inaccuracy. Training an AI model such as ChatGPT to judge the correctness of its own responses may result in better outcomes that adhere to medical guidelines. This innovation might entail creating a feedback loop in which the AI learns from expert judgments to improve the accuracy and safety of the solutions that it provides. Furthermore, given language accessibility, researchers might look into how ChatGPT could provide correct asthma information to underserved people in different languages, thereby boosting the utility of AI in public health.

Looking ahead, studies should encourage further research on the changing nature of the role of AI in health care communication. Future research efforts could include longitudinal studies that assess model improvements over time, as well as comparisons of the results provided by ChatGPT to peer-reviewed medical databases and literature. Furthermore, studying patient intake and comprehension of AI-generated health data may aid in better understanding the function of AI in doctor-patient dynamics and potential limitations as a communication tool in health care settings. The inclusion of a broader range of health subjects may also serve to spark discussions concerning the dependability of AI in conveying medical information.

Disclosure statement

Disclosure of potential conflict of interest: The authors declare that they have no relevant conflicts of interest.