Evaluation of the quality and readability of ChatGPT responses to frequently asked questions about myopia in traditional Chinese language

Abstract

Introduction

ChatGPT can serve as an adjunct informational tool for ophthalmologists and their patients. However, the reliability and readability of its responses to myopia-related queries in the Chinese language remain underexplored.

Purpose

This study aimed to evaluate the ability of ChatGPT to address frequently asked questions (FAQs) about myopia by parents and caregivers.

Method

Myopia-related FAQs were input three times into fresh ChatGPT sessions, and the responses were evaluated by 10 ophthalmologists using a Likert scale for appropriateness, usability, and clarity. The Chinese Readability Index Explorer (CRIE) was used to evaluate the readability of each response. Inter-rater reliability among the reviewers was examined using Cohen's kappa coefficient, and Spearman's rank correlation analysis and one-way analysis of variance were used to investigate the relationship between CRIE scores and each criterion.

Results

Forty-five percent of the responses of ChatGPT in Chinese language were appropriate and usable and only 35% met all the set criteria. The CRIE scores for 20 ChatGPT responses ranged from 7.29 to 12.09, indicating that the readability level was equivalent to a middle-to-high school level. Responses about the treatment efficacy and side effects were deficient for all three criteria.

Conclusions

The performance of ChatGPT in addressing pediatric myopia-related questions is currently suboptimal. As parents increasingly utilize digital resources to obtain health information, it has become crucial for eye care professionals to familiarize themselves with artificial intelligence-driven information on pediatric myopia.

Introduction

ChatGPT, a large language model (LLM) developed by artificial intelligence (AI) using supervised and reinforcement learning techniques, is a tool that generates human-like text responses to human input with high accuracy, accessibility, and low cost. ¹ Due to its accurate and well-formulated responses to various queries, ² ChatGPT has rapidly gained global attention. Of particular interest is its ability to grasp natural language in the field of healthcare. ³ ChatGPT has been used to support medical examinations, confirm clinical diagnoses, support clinical decision-making, and provide professional advice for patient education. ⁴ Due to its wide recognition and acceptance, ChatGPT is widely used to disseminate medical information to the general public. It can serve as a valuable tool for healthcare professionals in providing patient education and comprehensively addressing patient concerns. The evaluation of the responses of ChatGPT to health queries through grading by an independent physician for accuracy and reproducibility has gained considerable attention, paving the way for the potential integration of ChatGPT with patient education strategies.^5,6

The global prevalence of myopia, a condition that causes visual impairment, is expected to increase significantly to approximately five billion affected individuals in 2050. ⁷ Within educational systems emphasizing high academic achievement expectations, myopia is the most prevalent condition encountered in pediatric ophthalmology practices and has emerged as a public health issue among schoolchildren in Chinese-speaking societies with a prevalence far exceeding that observed in western countries.^7,8

Parents often lack knowledge about the diagnosis and treatment of myopia, and this presents challenges that may not always be fully addressed within the busy clinical settings of ophthalmologists. ⁹ Online discussion fora and websites are an easily accessible source of information for parents of myopic children. ¹⁰ Understanding parents’ discussions about myopia on the internet helps focus on their needs within the broader context of myopia issues. The comprehensibility and accuracy of the answers are significant parameters of appropriateness to address. ¹¹ Accurate online medical information enhances the comprehension of conditions and guides patients toward suitable treatments. Similarly, incorrect or misleading information can lead to confusion, delayed treatment, or the choice of detrimental “therapies.” Therefore, appropriateness and readability are key factors in health information provision. ¹² Health education information read by patients should be accurate, reliable, and current, whereas highly readable information is more accessible to readers. Consequently, the ability to apply information easily to address specific health issues is a core necessity for information usability. ¹³

A study of the responses of ChatGPT to 28 questions related to childhood myopia showed an acceptability rate of 87.5%. ¹⁴ Biswas et al. ⁶ demonstrated that ChatGPT generates responses to 11 myopia FAQs with an acceptability rate of >80%. In addition to professional assessment, the readability of responses as judged by analyzing their vocabulary and grammatical features using readability formulas is an important evaluation criterion. ¹⁵ The number of investigations evaluating the reliability of ChatGPT responses for patient education is increasing but the number of studies on ChatGPT responses to health issues in the Chinese language is limited. We aimed to evaluate the reliability and readability of childhood myopia-associated information provided to parents by ChatGPT in the Chinese language.

Methods

Data collection

This study collected public posts from a private Facebook group on pediatric myopia over one year from March 2022 to March 2023. Using keywords such as “myopia,” “diagnosis,” “treatment,” “atropine,” “deterioration,” “spectacle,” “improvement,” and “follow-up,” the top 20 most frequently asked questions in the published articles were identified according to the number of responses. The inclusion criteria were as follows: (1) posts in traditional Chinese; (2) issues related to the diagnosis, prevention, and treatment of childhood myopia; and (3) posts with responses by more than 100 individuals. The exclusion criteria included advertisements related to myopia, inquiries about treatment costs, and product recommendations (Table 1).

Table 1.

Appropriateness, usability, and clarity of ChatGPT responses to FAQs related to myopia.

	Question	Appropriateness			Usability			Clarity			All
		≥4	Median	IQR	≥4	Median	IQR	≥4	Median	IQR	≥4
1	Does a visual acuity of 1.0 indicate the absence of myopia?	10	5	5–5	10	5	5–5	10	5	5–5	10
2	Is cycloplegic refraction necessary during an eye examination?	10	4	4–5	10	5	4–5	9	4	4–5	10
3	Can short-acting mydriatic agents control the progression of myopia?	2	3	2–3	2	3	2–3	1	2	1–3	1
4	When does my child need to wear glasses for myopia?	3	3	3–4	3	4	2–4	3	3	2–4	2
5	Do atropine agents cause cataracts?	1	3	1–3	1	2	1–3	0	1	1–2	0
6	What is the recommended age for using atropine eye drops for the treatment of myopia?	9	5	4–5	7	4	3–5	9	4	4–5	6
7	What are the evidence-based methods for treating myopia in children effectively?	7	4	3–4	5	4	3–5	4	4	3–4	4
8	Is Orthokeratology (Ortho-k) corneal reshaping lens safe?	9	5	4–5	9	5	4–5	9	5	4–5	9
9	Can laser treatment be used to treat myopia?	3	4	3–4	3	4	2–4	1	3	2–3	2
10	Can wearing glasses control myopia?	6	4	4–5	5	5	4–5	5	4	4–4	4
11	What precautions should I take when using atropine eye drops?	9	5	4–5	6	5	3–5	10	5	5–5	6
12	Will discontinuing atropine eye drops cause a rebound in myopia progression?	6	4	4–5	9	5	4–5	7	5	3–5	6
13	Is blue light filtering eyewear effective in controlling myopia?	7	5	4–5	8	5	4–5	6	5	4–5	7
14	Is it necessary to use atropine eye drops daily?	10	5	5–5	10	5	4–5	10	5	5–5	10
15	Does receiving orthokeratology eliminate the need for atropine eye drops for treatment?	7	4	5–5	7	5	4–5	7	4	4–5	6
16	Can vision training treat childhood myopia?	9	5	5–5	9	5	4–5	9	5	4–5	7
17	What data should I look for to determine if myopia is not worsening?	7	4	4–5	8	5	4–5	7	4	3–5	8
18	What are the effective methods for preventing myopia?	10	5	5–5	10	5	4–5	10	5	4–5	10
19	Is it necessary to use low-dose atropine eye drops for the prevention of myopia?	9	4	4–5	7	5	3–5	6	4	4–5	9
20	Does not wearing glasses worsen the degree of myopia?	2	3	3–4	3	3	2–4	3	3	2–4	3

Note “all” refers to the number of experts who rated appropriateness, usability, and clarity all above 4.

Twenty questions standardized by the researchers were selected and classified into six categories: (1) diagnosis; (2) atropine; (3) spectacles; (4) orthokeratology; (5) effective control, and (6) others (Table 2).

Table 2.

Performance and comparison of ChatGPT responses across categories.

Categories	No.	CRIE scores	CRIE	Appropriateness	Usability	Clarity
			Mean (SD)	ANE (SD)	ANE (SD)	ANE (SD)
1. Diagnosis	1	9.78	9.71 (0.10)	10.00 (0.00)	10.00 (0.00)	9.50 (0.70)
	2	9.63
2. Atropine	5	9.21	8.76 (0.87)	6.67 (3.314)	7.33 (3.38)	7.00 (3.79)
	6	9.25
	11	8.05
	12	9.88
	14	7.50
	19	11.67
3. Spectacles	4	7.29	7.34 (0.47)	3.67 (1.15)	3.67 (2.08)	3.37 (1.15)
	10	7.36
	20	7.38
4. Orthokeratology	8	8.00	8.13 (0.18)	3.50 (2.12)	4.50 (3.53)	3.55 (2.12)
	15	8.26
5. Effective treatment	3	9.38	9.46 (1.08)	8.00 (1.41)	8.00 (1.14)	8.00 (1.41)
	7	9.53
6. Others	9	11.46	10.76 (1.06)	7.60(2.70)	7.20(2.68)	6.60(3.50)
	13	9.29
	16	10.67
	17	12.07
	18	10.31
F(p)	—	—	5.71(0.004)	2.462(0.080)	1.69(0.200)	1.67(0.200)

Note no: the item number of the FAQs; ANE: the average number of experts who rated the questions above 4; SD: standard deviation; CRIE: Chinese Readability Index Explorer; F(p) indicates the ANOVA results for CRIE scores and expert review scores.

ChatGPT response generation

ChatGPT (version GPT-3.5, OpenAI) was queried three times (i.e. in three separate “trials”) ¹⁶ for each question in traditional Chinese language on 16 April 2023. An example of a prompt entered into the ChatGPT interface is “I am a parent of an elementary-school student, and I would like to know how myopia is diagnosed?” To prevent the effect of interaction between successive inputs, each query was run on a fresh ChatGPT session. Moreover, each prompt was regenerated three times, with each answer rated separately to assess the variation in response quality.

Quality evaluation based on expert reviews

We invited ophthalmologists who met the following criteria: (1) having more than eight years of experience in pediatric myopia; (2) dedicating at least one pediatric myopia outpatient clinic per week; and (3) having published case reports or studies related to pediatric myopia. Ten experts participated and rated the appropriateness, usability, and clarity of each response using a five-point Likert-type scale from 1 (very inappropriate/very difficult to use/extremely unclear) to 5 (very appropriate/usable/clear), as previously described. ¹⁷ The inter-rater agreement between the 10 ophthalmologists was excellent with a Kappa value of 0.820 (95% confidence interval, 0.721–0.920).

We calculated the median, interquartile range (IQR), and range of scores for each question. For scores less than 4, ophthalmologists provided explanations for the answers deemed inappropriate. Only responses that obtained scores of ≥4 on all three indicators (appropriateness, usability, and clarity) by at least eight ophthalmologists were considered satisfactory.

Analysis of readability using the Chinese readability index explorer

The readability of each response was assessed using the Chinese Readability Index Explorer (CRIE, version 3.0, available at http://www.chinesereadability.net/CRIE), an online automated tool for analyzing Chinese text. ¹⁸ The CRIE integrates four subsystems and 82 multilevel linguistic features to analyze word count, lexical richness, semantic complexity, syntactic diversity, and cohesion. Recent studies have confirmed the reliability of the CRIE in assessing the readability of texts written in Chinese.^19,20 In this study, we entered 20 responses into the CRIE system to create readability prediction models and assign scores to categorize text into the corresponding grade reading levels: levels 1–6, elementary school (grades 1–6); levels 7–9, middle school (grades 7–9); and levels 10–12, high school (grades 10–12).

Statistical analysis

We conducted a descriptive analysis of the scores by 10 ophthalmologists, including the median and the IQR. Cohen's kappa coefficient was used to measure inter-rater reliability. One-way analysis of variance (ANOVA) was used to test for differences in the readability and expert ratings of the categorized responses. Spearman's rank correlation analysis was performed for appropriateness, usability, clarity, and CRIE scores, with the significance level set at p < 0.05.

Results

As shown in Table 1, the median appropriateness, usability, and clarity scores of the responses to the 20 FAQs ranged from 3 to 5, 2 to 5, and 1 to 5, respectively. Five of the responses (25%) met the satisfactory response criteria as rated by more than eight ophthalmologists. These satisfactory responses covered topics such as “significance of visual acuity (#1),” “necessity of cycloplegic refraction to diagnose myopia (#2),” “corneal reshaping mechanisms of orthokeratology (#8),” “indicators of myopia progression (#17),” and “effective methods for preventing myopia (#18).” For example, the highly rated response for topic #8 was “Orthokeratology (Ortho-K) corneal reshaping lenses are a type of specialized contact lenses that can correct myopia by reshaping the cornea. Compared to traditional contact lenses, Ortho-K lenses have several advantages, such as effectively controlling the progression of myopia and eliminating the need for daytime glasses or contact lenses.”

Responses related to treatment efficacy (#3, #4, #9, and #20) and side effects (#5) were rated low for all three criteria (no more than three ophthalmologists provided scores of >4) and were considered inaccurate or flawed by the ophthalmologists. These responses included suggestions of short-term efficacy for myopia treatments, wearing spectacles, recommendations for prolonged use despite pupil dilation, and endorsement of laser for myopia. For example, a poorly rated response was for topic #9: “Laser treatment can be used to treat myopia, primarily through two methods: LASIK and LASEK/PRK. Both procedures require a longer recovery period and care. Regardless of the method used, laser treatment for myopia should be performed by experienced professionals in appropriate medical facilities. Detailed evaluation and examination are necessary to determine the suitability and safety of the treatment plan.”

Table 2 presents the mean scores of items and categories for CRE, along with the average number of experts who rated the questions above 4. The CRIE readability scores for the ChatGPT responses ranged from 7.29 to 12.07. The readability of 35% of the responses (to seven questions) was at the middle-school level, while that of the remaining questions was above the high-school level (>9). Responses related to laser surgery (#9), low-concentration atropine for myopia prevention (#19), and determination of myopia progress (#17) had high readability scores (>11). When categorized by the nature of the questions, responses about spectacles and orthokeratology had relatively lower readability scores (mean = 7.34 and 8.13, respectively), while those discussing effective treatments and diagnoses had higher readability scores (mean = 9.46 and 9.71, respectively).

As shown in Table 2 and Figure 1, the results of one-way ANOVA showed significant differences in the readability of the ChatGPT responses to different categories of FAQs (p = 0.004), whereas no significant differences were observed among the expert evaluations on all three criteria. Table 3 shows the correlations between the CRIE readability scores and expert review scores. Spearman's rank correlation analysis showed no significant correlations between the expert ratings of appropriateness, usability, and clarity of the responses and the CRIE readability scores.

Figure 1.

One-way of analysis of variance of different treatment categories for myopia management.

Note: Category: 1. Diagnosis, 2. Atropine, 3. Spectacles, 4. Orthokeratology, 5. Effective treatment, 6. Others.

Table 3.

Correlations between CRIE, appropriateness, usability, clarity.

Variables	CRIE	Appropriateness	Usability	Clarity
CRIE	1	0.184	0.201	0.244
Appropriateness		1	0.711**	0.854**
Usability			1	0.897**
Clarity				1

Note ** p < 0.01.

Discussion

This study is the first to demonstrate that a novel LLM-derived conversational AI program can provide appropriate, usable, and clear responses in traditional Chinese language to myopia-related FAQs, as determined by ophthalmologists and the CRIE. ChatGPT is an AI model based on deep learning and natural language processing, and it is trained on vast amounts of text to generate human-like text responses by predicting the most probable next word or sequence of words when given a prompt or question. ¹⁷

This study found that less than half of the ChatGPT myopia-related responses in the Chinese language met the criteria for appropriateness, usability, and clarity, as evaluated by ophthalmologists. Several studies have highlighted the suboptimal clarity of online or printed materials related to ophthalmic diseases, which impairs patient comprehension.^21,22 The reports on the quality of ChatGPT responses to FAQs by previous studies have been inconsistent.^14,23 ChatGPT provided high-quality responses to questions from the general public, as evaluated by physicians, ³ while its performance in addressing ophthalmologic treatment questions was low.^22,24,25 This is consistent with the findings of the present study.

Nikdel et al. ¹⁴ reported that more than 80% of the responses of ChatGPT 4.0 to questions about myopia were deemed appropriate by two ophthalmologists; however, the percentage of appropriate responses in this study was lower. In addition to using different versions of ChatGPT, this difference may be attributed to the number of ophthalmologists invited to evaluate the responses and the language used. In the study of Biswas et al., ⁶ the response of ChatGPT to the question of whether a single treatment could effectively control myopia was assigned high accuracy scores by experts. Similar to Lim and Pushpanathan, ²⁰ we asked ChatGPT about each specific myopia treatment, and the responses to each question in our study received lower appropriateness ratings. The use of atropine eye drops is more common in Asian countries with high prevalence of myopia, such as Hong Kong, Taiwan, and mainland China, whereas it is less commonly studied in the United States. ²⁶ The quality of the responses of ChatGPT about the use of atropine was less satisfactory in our study. This indicates that the primary training texts for ChatGPT are in English, ²⁷ and its responses tend to align with the American socio-cultural environment. This may limit its adaptability to other cultural contexts, especially in understanding and addressing treatments more commonly used in Asia. ²⁸

The readability of health information is assessed based on factors such as sentence length and average number of syllables per word to determine the reading level required to understand the text effectively. ²⁹ Based on the CRIE readability scores, the responses related to spectacles and orthokeratology were rated at middle-school level. This is consistent with the findings of King et al. ³⁰ who demonstrated that the readability of ChatGPT responses exceeds the American Medical Association's recommended reading level of fifth to sixth grade. Consistent with the findings of Tao and Hua (2022), ³¹ the readability level of ophthalmology-related ChatGPT responses also exceeded that of eighth grade. The expert ratings for spectacle-related responses were low, whereas responses concerning orthokeratology were rated more favorably in terms of expertise. The readability of the responses of ChatGPT is influenced by the quality of its training sources ²⁸ and the prompts provided, which should be tailored to the reader's literacy level. ³² Given that the training data of ChatGPT originate from existing sources, it is plausible that materials regarding spectacles were not composed by ophthalmologists. This may contribute to perceptions of insufficient professional depth from an ophthalmological viewpoint. Additionally, ophthalmologists may maintain conservative views on the role of spectacles in treating myopia, potentially leading to disagreements with responses linking spectacles to myopia treatment. ³³ Orthokeratology is supported by extensive evidence-based studies, ³⁴ practical guidelines, ³⁵ and comprehensive health education materials,^36,37 providing ChatGPT with robust textual resources for more accurate responses.

The responses of ChatGPT to inquiries related to the diagnosis and effective treatment of myopia have a high readability level, often necessitating a reading proficiency beyond that of the middle school. This high level of readability highlights the need to pay more attention to the lower level of knowledge displayed by parents concerned with these topics. ¹⁰ The study identified a lack of significant correlation between readability and the quality of information provided by ChatGPT, corroborating earlier findings that the quality of health information does not necessarily correspond with its readability. ³⁸

The performance of ChatGPT in addressing mydriatic treatments for myopia was inadequate and potentially misleading. The appropriateness was significantly lower, warranting further attention. This resonates with the notion that ChatGPT does not possess genuine comprehension or knowledge of the text it generates, and that a lack of deeper understanding of the context or meaning behind the words it uses could lead to misleading responses.

This study has some limitations. First, the 20 FAQs were obtained from websites, rather than directly gathering queries from parents during clinical visits. This may have limited the representation of their genuine concerns. Second, interaction with ChatGPT was confined to a single prompt, potentially constraining the precision of the responses obtained. Future research should employ diverse prompts and other versions of ChatGPT or Gen AI machine to verify responses. Third, this study evaluated the quality of the responses of ChatGPT based solely on ophthalmologists’ opinions. Future analyses should include other medical resources and revise any inaccuracies in the responses of ChatGPT based on expert feedback to provide more accurate and useful information.

Eye care practitioners may not be easily accessible, and their services may be costly and time-consuming. Conversely, online chatbots have gained widespread popularity and attention among parents. AI is viewed as a promising tool, offering opportunities to explore its utility in patient interactions and providing valuable information on eye diseases. In the future, families, schools, and pediatric nurses should collaborate to assist parents in evaluating information related to the prevention and treatment of myopia using ChatGPT.

Given the significance of online information as a pivotal channel for the parents of children with myopia in guiding treatment decisions,^10,39 it is essential for ECPs to comprehend the role and functionality of ChatGPT in health education for myopia control. Based on the findings of this study, several recommendations can be made. First, to better utilize ChatGPT in healthcare services, healthcare professionals need to receive training on effective prompting techniques to engage with ChatGPT across various topics, focusing on formulating appropriate questions to obtain more reliable information through multi-step interactions. Additionally, professionals should be aware of the language and cultural differences in ChatGPT's training data to avoid potential biases or misleading information. This study utilized ChatGPT-3.5 with one-shot prompting to generate responses. Future studies should compare the appropriateness, usability, and clarity of responses from different ChatGPT versions, other generative AI models, and prompting techniques to optimize the use of ChatGPT as a tool for supporting healthcare professionals.

Conclusions

In summary, this study evaluated the performance of ChatGPT in responding to the most commonly asked myopia-related questions by parents online. ChatGPT performed better in addressing general myopia-related questions than in responding to treatment-related questions, particularly those concerning the use of atropine eye drops and spectacles for children. The responses of ChatGPT in this study demonstrated high appropriateness, usability, and clarity regarding atropine treatment, making ChatGPT a valuable supplementary tool for patient and health education. Importantly, education materials should remind parents to ask additional questions to verify the accuracy of ChatGPT's responses concerning eyeglasses. However, the readability of the responses provided by ChatGPT to myopia-related questions still exceeded the American Medical Association's standard of being below the sixth-grade level. Healthcare professionals need to be trained in the use of generative artificial intelligence to ensure they can appropriately identify the accuracy of myopia-related health information generated by ChatGPT and align it with the comprehension levels of individual patients.