ChatGPT in the development of medical questionnaires. The example of the low back pain

Abstract

In the last year, Chat Generative Pre-Trained Transformer (ChatGPT), a web software based on artificial intelligence has been showing high potential in every field of knowledge. In the medical area, its possible application is an object of many studies with promising results. We performed the current study to investigate the possible usefulness of ChatGPT in assessing low back pain. We asked ChatGPT to generate a questionnaire about this clinical condition and we compared the obtained questions and results with the ones obtained by other validated questionnaires: Oswestry Disability Index, Quebec Back Pain Disability Scale, Roland-Morris Disability Questionnaire, and Numeric Rating Scale for pain. We enrolled 20 subjects with low back pain and we found important consistencies among the validated questionnaires. The ChatGPT questionnaire showed an acceptable significant correlation only with Oswestry Disability Index and Quebec Back Pain Disability Scale. ChatGPT showed some peculiarities, especially in the assessment of quality of life and medical consultation and treatments. Our study shows that ChatGPT can help evaluate patients, including multilevel perspectives. However, its power is limited, and further research and validation are required.

Ethical Publication Statement

We confirm that we have read the Journal’s position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

During the last year, one of the potential greatest revolutions impacting humankind has been happening: the worldwide availability of chatbots based on artificial intelligence (AI).¹ In particular, on November 30th, 2022, Chat Generative Pre-Trained Transformer (ChatGPT) was released.² It is an online software able to communicate in a human-similar way and to answer specific questions. Furthermore, as an AI system, it can learn from the information that the user provides and dynamically interact with the human being.2 This type of chatbot distinguishes itself from simple web-based software because of its adaptability. Indeed, it can tailor its responses to various situations and to learn from errors and diverse inputs. This capability is advantageous for solving a wide range of problems by retrieving information from a vast dataset. Its practical applications are highly diverse and may virtually include every field of knowledge. For work activities, it can solve equations or help in computer programming; for leisure time, it can provide suggestions about a trip or invent a game; for daily life, it can support text or email writing; and so on.³

It has, at the moment, different limitations, due to the information that the ChatGPT programmers used to train it, covering a time range until the end of 2021, and to the immaturity of some procedures. For example, ChatGPT has some difficulties in elaborating very long or highly specific texts. Additionally, too complex tasks can lead to errors and the accuracy of the responses depends on the precision of the questions.1 Despite these current limitations, the chatbot has demonstrated remarkable potential as an assistive technology and our comprehension of its capabilities is probably only in the early stages. In the few months of its availability, even in the medical field, we have been observing a growing utilization with extraordinary outcomes. The tool can combine different information from literature and scientific databases, contributing to the interpretation of research findings or the discovery of new solutions for projects.

Various studies have identified potential roles for chatbots in supporting physicians' training, particularly in summarizing patient variables and extracting relevant information from extensive datasets, like big data analysis. Moreover, for medical students, chatbots can assist in research, essay writing, or thesis work.⁴ In the medical field, this AI can also support patient-physician communication, simplifying medical explanations with everyday language, assessing and consolidating symptom descriptions, and facilitating translations of medical data into various languages.⁵ ChatGPT can also be beneficial for patients, providing suggestions for a health or dietary plan or indicating the most appropriate medical specialist.⁶ Importantly, the chatbot underscores its restricted role in its responses, emphasizing the importance of the consultation of an irreplaceable human expert in cases that require specialized medical attention.

Besides the incredible positive impact on the work and the study, the risks of misunderstood are always behind the corner. Additionally, the peril of dehumanization of medical activity and the tendency to reduce the effort toward effective work are the potential side effects of the increasing use of AI. Moreover, the utilization of this tool for certain tasks raises complex ethical concerns, like the production of scientific items sold as authors’ original production but really made by a computer.⁷

Therefore, currently, ChatGPT cannot replace human doctors and scientists.⁸ Nonetheless, we should avoid prejudicial judgment and we need to study all the main real effects that the chatbot may produce. Considering its medical applications, limited examples may be found in questionnaire assessments or in constructing new specific questionnaires. The existing questionnaires for understanding patients' perspectives on diseases are well-established, validated, and easy to administer. However, there is a risk of incompleteness in these questionnaires. Furthermore, as patient needs continually evolve with societal changes, updating historical questionnaires becomes desirable.

Typically, questionnaire development starts with collecting qualitative or semi-quantitative data from a large database obtained through a survey.⁹ The first problems in this procedure are related to the choice of the initial questions and the answer options used for the survey, the language, and, especially, the lexicon.¹⁰ Even the communication channel and the correct choice of the interviewed people may greatly affect the answers. After data collection, a simplification process is necessary to construct the actual questionnaire, which must undergo testing for validity and reliability.9 Once a questionnaire passes these stages, additional processes of cross-cultural validation and adaptation are needed for translation and application in cultures and languages different from those for which the questionnaire was originally developed.9 This brief description of the intense phases of questionnaire development suggests the possible aid of ChatGPT for this purpose. The chatbot can combine data from other existing questionnaires, and scientific and not academic information from the entire world, and can translate the document with precision.

Building upon these considerations, we aim to create a questionnaire for the assessment of [low back pain (LBP)] using ChatGPT and to evaluate its effectiveness by comparing it with routinely used validated questionnaires.

Materials and Methods

We performed an observational cross-sectional study. We enrolled subjects with a history of LBP of different severity in the last six months. Exclusion criteria were: age lower than 25 years old; recent history of trauma involving the spine; previous orthopedic, rheumatologic, and neurological diseases impacting the function of the musculoskeletal system; recent oncologic conditions with metastases; use of drugs influencing the metabolism of the bone. In order to build the questionnaire, we asked ChatGPT 3.5 to generate a questionnaire for the assessment of the LBP. The questions were formulated in Italian and the answers of the software were delivered in the same language (Supplementary file). The choice of the Italian language was determined by the necessity to test it with an Italian-speaking group of patients. ChatGPT questionnaire (ChatQ) consisted of ten questions exploring different aspects of the LBP from different points of view. In six questions the evaluation was focused on the pain perceived and its worsening factors. One question was directly aimed at the investigation of LBP impact on the quality of life (QoL). The remaining three questions explored the treatments and the medical management of LBP. Because of the different areas explored, to compare the results of ChatQ with the other tools validated in the Italian language, we consolidated the results of some questions produced by the AI to obtain a final numerical value. In particular, the single-choice and ordinal answers (for example, never, sometimes, often, always) were transformed into numerical values (in the same example, from 0 to 3). The first three questions, assessing frequency, intensity, and duration of the pain and the question about QoL were treated with this simplification. The answer about the extension of the pain was considered as a binary value (0 if the pain was located in one site, 1 if it was diffuse). The answer indicating the worsening factors was transformed in value 1 if one activity was reported and in value 0 on the other hand. The answers showing the additional symptoms were simply converted into a number indicating the count of the reported symptoms (from 0 to 4). The answers about QoL were converted into: 0 (no impact), 1 (low impact), 2 (high impact). These numbers were summed to obtain the final score of the ChatQ (possible range 0–17). The other three questions about the medical treatments were not included in the final score. The questionnaire was printed and administered to the subjects for self-compilation. The subjects were also asked to complete three validated questionnaires for back pain: Oswestry Disability Index (ODI), Quebec Back Pain Disability Scale (QBPDS), and Roland-Morris Disability Questionnaire (RMDQ). The ODI is a self-administered and Italian-language validated questionnaire designed to assess disability in individuals with LBP in both the acute and chronic phases. The test is considered the “gold standard” of low back functional outcome tools and investigates how LBP affects different aspects of daily life. Each section contains within it 6 possible answers (from 0, no disability, to 5, maximal disability). The final score is expressed as a percentage of disability (from 0% to 100%).¹¹ The QBPDS measures the difficulties in everyday activities for people with LBP. The questionnaire is composed of 20 items and each activity is scored on a Likert scale from 0 to 5 (0 = no difficulty and 5 = total inability). The final score can range from 0 to 100.¹² The RMDQ is designed to assess self-rated physical disability caused by LBP. The patient is instructed to put a mark next to each appropriate statement. The final score corresponds to the total number of marked statements, with a range from 0 to 24.¹³ Finally, a generic evaluation of the pain, using the numerical rating scale (NRS), in which patients are asked to circle the number between 0 and 10, was applied.¹⁴ General clinical data about gender, age, and body mass index were also collected. All data were gathered in anonymous form and the questionnaire inspector was blind to the patients' answers. This study was conducted in adherence to the principles outlined in the Declaration of Helsinki. The enrolled subjects signed an informed consent of their data treatment. The final scores of ChatQ (as described) and of the other instruments were summarized with median values. Additionally, the ordinal answers of ChatQ (frequency, intensity, duration, presence of other symptoms, impact on QoL) were summarized as median values and shown as bar charts. The binary answers (yes or no) about the extended localization of the pain, the presence of worsening factors, and the medical consultation and treatments were summarized as frequency values and pie charts. Finally, a bar chart was used to show the number of times a treatment was reported. The final scores of the various questionnaires were compared by Spearman's correlation analysis, because of the type of the data. Statistical significance was set as p < 0.05. The analysis was performed by the freeware software Jamovi 2.3.18. Finally, a comparison of the different main areas explored by the different questionnaires was performed, studying the most common words used in the questions and answers, by the use of TXM 0.8.0 freeware software.¹⁵

Fig 1.

Results of the artificial intelligence questionnaire. The bar charts indicate the median value of the ordinal variables and the number of treatments. The pie charts indicate the frequency of “yes” and “no” in the answers to the other questions.

Results

A total of 20 subjects were enrolled (13 women, median age of 41 years, age range of 27–78 years, and median body mass index of 23.41 kg/m2). All the patients were able to answer the administered questionnaires without assistance. The questions generated by the AI did not request specific explanations or adaptations. Median values and minimum-maximum results of the questionnaires were the following: ChatQ 8/17 (4/17–12/17); ODI 12% (0%–42%); QBPDS 9/100 (0/100–45/100); RMDQ 3/24 (0/24–11/24); NRS 4/10 (0/10–8/10). Considering the single items of the ChatQ, no subjects referred absence of LBP, in one single case the reported frequency was “always”, and the median value of this item was “often”. The median intensity score was classified as moderate in two cases and as severe in another. The median reported pain duration was 4-6 hours, but for eight subjects it was more than 6 hours. Three additional symptoms (paresthesia, lower limb pain, and neck pain) were observed in a single case, compared to a median value of one, and the most frequent location being the neck. Only two patients considered their LBP severely impacted their QoL, while the low impact on QoL represented the median value. In 25% of the cases, the patients described their pain as extended in their whole trunk, and 90% of the cases reported worsening factors, mostly consisting of prolonged positions. In 55% of the cases, a medical consultation was requested; however, only 45% of the responses disclosed the medical treatments. Lastly, medication was the most commonly prescribed treatment, followed by massage and physiotherapy. Although the ChatGPT questionnaire considered chiropractic as a medical treatment, no patient admitted the use of this approach (Figure 1). The correlation analysis showed high statistical significance with strong to very strong correlation degrees among ODI, QBPDS, and RMDQ and between this latter and NRS. The correlation between NRS and ODI or QBPDS was statistically significant but of moderate level. A strong significant correlation was found between ODI and ChatQ and a moderate correlation between QBPDS and ChatQ. No statistical correlations were present between ChatQ and RMDQ or NRS. In both cases, the scatterplots showed a tendency to a consistent mutual increase of the two analyzed variables (Figure 2). The main semantic fields investigated by the various questionnaires showed common families of elements related to pain. In particular, in all the questionnaires, including ChatQ, specific questions about actions and/or movements that can impact the pain were administered. In QBPDS, the assessment of sleep quality was added, while ODI and RMDQ additionally included questions about social relations. Only in ChatQ, specific questions about the medicalizations and the treatments were present (Figure 3).

Fig 2.

Scatterplots of the total scores of the different questionnaires. The tendency line is illustrated and the values of “r” and “p” are indicated.

Discussion

Our results suggest that ChatQ, specifically designed for assessing LBP, can play a potentially effective role in clinical evaluations. The results were consistent with a low level of disability in most of the subjects, while, in three subjects, the LBP was moderately severe, according to the usual validated scales. ChatQ demonstrated itself as a reliable and valuable tool for assessing LBP. Indeed, ChatQ demonstrated some good correlations with established LBP measures and allowed to explore a relatively large range of facets related to pain. Considering the relationships among the five studied tools, as expected, the four validated instruments (ODI, QBPDS, RMDQ, and NRS) showed a robust level of correlation.¹⁶ ChatQ presented a good correlation with ODI and an acceptable one with QBPDS, but no significant results were found for RMDQ and NRS. Therefore, the precision of ChatQ as a quantitative measure is probably low. A possible explanation for the absence of correlation with NRS may be due to the different types of pain characterization. NRS is focused on the intensity, while ChatQ adds information about frequency and duration. In fact, ChatQ covers a wide range of topics regarding pain perception, such as intensity, frequency, and duration. In clinical practice, the relation between these three components may be very helpful in defining the severity of LBP for both the patient and the clinician. It is possible to better define the nature and impact of pain on daily activities by focusing on all of its characteristics rather than just its intensity. Moreover, ChatQ, like other questionnaires, investigates the location of the pain and whether worsening factors are present. Different established protocols have been proposed to report pain by drawing on various body-representation templates.¹⁷ Nevertheless, ChatQ introduces the possibility of indicating which portion of the back is affected, better defining the patient’s suffering.¹⁸ Besides the interrogations strictly focused on pain features, ChatQ explores the impact on QoL with a precise and direct question. As QoL and, in general, the impact of LBP on daily activities are two of the most important outcomes of rehabilitation, the inclusion of direct questions about QoL may help to address the evaluation.¹⁹ However, this simple approach may not be precise enough to assess the complexity of QoL. Indeed, a well-designed questionnaire should investigate both the mental and physical aspects of life to ascertain which area is more impacted. ChatQ also queries whether there are any additional symptoms to the primary discomfort. This data could provide significant clinical information that could help define and treat the pain more precisely. Obviously, this particular information should only be used as a helpful tool in its current form. For example, the presence of paresthesia, if reported, should guide the physician to administer further questionnaires or complete the clinical examination to depict the eventual presence of neuropathic pain.15 ChatQ goes beyond existing questionnaires by including inquiries about medical management, which offers an innovative perspective. The three questions on this topic in ChatQ can reveal multiple facets of the patient's experience. First, the presence or not of the medical consultation may be related to the predisposition of the patient to refer to a doctor. This is a relevant point because may be a “litmus test” of the patient-physician relationship for the specific subject and may support the construction of a proficient dialogue.²⁰ It also provides insights into how patients perceive the severity of their pain and their health status. Furthermore, this question should be interpreted in conjunction with the inquiry about prescribed medical treatments. Indeed, medical consultation does not imply compulsory medical treatment. The absence of a prescribed medical treatment could indicate a milder condition, where a physician may adopt a wait-and-see approach or suggest minor lifestyle changes, if the back pain is not highly debilitating.²¹ However, these data must be complemented by responses about the specific referred medical treatments. These answers reveal an interesting behavior of the subjects. Although a medical consultation was not reported and no medical treatments were indicated, some subjects revealed the use of self-administered therapies, in particular drugs. This information is in accordance with the vast worldwide literature that shows the pandemical problems of auto-treatment, with the several potential side effects and the risk of misdiagnosis and difficulties, in general, of proper management.²² Conversely, in some cases, when patients mentioned physiotherapy or massage, no medical treatment was recorded, possibly because the patients did not perceive them as medical interventions. This point should be better investigated with further and more precise surveys. Interestingly, ChatQ distinguishes between massage and physiotherapy. This may be interpreted with a high level of knowledge of the computed system. Although not totally correct, this distinction possibly underlines the differences between a basic manual therapy and a more specialized one.²³ Furthermore, the AI-generated questionnaire includes complementary and alternative medicines as possible treatment strategies.²³ Among our participants, one mentioned acupuncture, while no one referred to chiropractic care. These answers again highlight the capability of the AI to use sources from a vast database. If acupuncture has been demonstrating evidence in particular for pain control, chiropractic, at the moment, does not show a similar robustness.^24,25 However, the possibility of investigating these adjunctive traits may support the construction of new wider, and more complete questionnaires. Despite many worries about the rapidly developing new technologies, previous research has highlighted that generative AI implications in clinical practice and research are generally positively perceived among physicians in the fields of radiology, rheumatology and orthopedics.²⁶ Taken together, our results suggest that the questionnaire proposed by ChatGPT presents both potential advantages and limitations. Among the advantages, the evaluation of different pain variables can facilitate the identification of pain severity, while the previously validated questionnaires discriminate less clearly among these variables. AI-driven questionnaires can provide a standardized and consistent approach to LBP assessment, reducing potential variations that can occur in human-administered assessments. This standardization not only guarantees the homogeneity of criteria used to evaluate the patients, but also allows for the collection of more consistent data over time. A standardized data collection may help the rapid identification of potential issues, leading to quicker diagnosis and intervention for patients with LBP.²⁷ Undoubtedly, another advantage is related to the evaluation of QoL, although this should be further investigated to assess the various components of this item. Finally, the unique opportunity to investigate the relationships between the patient and medical activity deserves further detailed studies. On the other hand, the ChatQ questionnaire has significant limitations, primarily because it lacks questions about social and sexual relationships, which are crucial for assessing QoL.²⁸ Additionally, it does not cover sleep quality.²⁹ The absence of these important factors reduces the questionnaire's ability to provide a comprehensive evaluation of an individual's well-being. In the future, the inclusion of these items can improve the effectiveness of the AI-based questionnaire.

Fig 3.

Words in common among the different questionnaires..

In conclusion, while still immature, ChatQ shows great potential and is quite fascinating. Despite the presence of some significant correlations with other tools, it cannot replace them. However, it can be employed to explore new subfields in the clinical evaluation of patients, due to its ability to use various types of data. Its capacity to combine data in a human-like manner may simplify the development of evaluation tools, starting with the identification of specific topics. Additionally, AI can enhance the depth of assessment processing and analysis of a wide range of data sources, such as patient-reported information, medical records, and even imaging data. Thus, ChatGPT may be used to generate, with proper input, new integrated and specific questionnaires combining the questions and the information of other tools. Therefore, while our findings indicate the promise of AI-powered questionnaires for LBP, further research and validation are necessary to ameliorate their utilization and address potential limitations. The capabilities of this type of chatbot are likely only beginning to be imagined, and further research is essential to assess its applicability for patients’ benefits.

List of acronyms

AI

artificial intelligence

ChatQ

ChatGPT questionnaire

LBP

Low back pain

NRS

Numerical rating scale

ODI

Oswestry Disability Index

QBPDS

Quebec Back Pain Disability Scale

QoL

Quality of life

RMDQ

Roland-Morris Disability Questionnaire

Funding Statement

Funding: None to declare.