Evaluation of Patient Education Materials From Large-Language Artificial Intelligence Models on Carpal Tunnel Release

Abstract

Background:

ChatGPT, an artificial intelligence technology, has the potential to be a useful patient aid, though the accuracy and appropriateness of its responses and recommendations on common hand surgical pathologies and procedures must be understood. Comparing the sources referenced and characteristics of responses from ChatGPT and an established search engine (Google) on carpal tunnel surgery will allow for an understanding of the utility of ChatGPT for patient education.

Methods:

A Google search of “carpal tunnel release surgery” was performed and “frequently asked questions (FAQs)” were recorded with their answer and source. ChatGPT was then asked to provide answers to the Google FAQs. The FAQs were compared, and answer content was compared using word count, readability analyses, and content source.

Results:

There was 40% concordance among questions asked by the programs. Google answered each question with one source per answer, whereas ChatGPT’s answers were created from two sources per answer. ChatGPT’s answers were significantly longer than Google’s and multiple readability analysis algorithms found ChatGPT responses to be statistically significantly more difficult to read and at a higher grade level than Google’s. ChatGPT always recommended “contacting your surgeon.”

Conclusion:

A comparison of ChatGPT’s responses to Google’s FAQ responses revealed that ChatGPT’s answers were more in-depth, from multiple sources, and from a higher proportion of academic Web sites. However, ChatGPT answers were found to be more difficult to understand. Further study is needed to understand if the differences in the responses between programs correlate to a difference in patient comprehension.

Introduction

ChatGPT (OpenAI) is an online Web site that uses natural language processing and artificial intelligence to provide human-like responses to user queries. ¹ With 100 million users in February 2023, ChatGPT is very popular and has the potential to influence patients’ understanding of orthopedic pathology, surgical options, and postoperative recovery. Patients have long used online search engines such as Google Web Search, to learn about medical topics; however, the accuracy of these answers is not validated and may not always be generated from credible medical sources. With the commercialization of artificial intelligence that uses large language models, many patients now have started using other resources aside from traditional search engines for routine health-related questions. ²

Carpal tunnel syndrome (CTS) is an extremely common disease in the United States with an estimated prevalence of 1% to 5% of the general public having CTS confirmed on electrodiagnostic studies. ³ In addition, the lifetime prevalence of carpal tunnel release surgery was 3.1%, and the incidence rate was 1.73 per 1000 person-years. ⁴ In addition to receiving information from their physician, patients have the ability to perform their own research using online resources.^{5 -7} Kwak et al investigated informational YouTube videos related to carpal tunnel surgery and found that they had both low-quality and low-reliability information. ⁸ In another study, frequently asked questions (FAQs) on Google were found to have a low degree of transparency and quality. ⁹ These studies indicate the need for an improved system that allows patients to ask questions and perform self-directed research into their disease. Although there are multiple studies examining Google’s responses to carpal tunnel release surgery questions, there is limited data comparing this database to other forms of online information.

Currently, the American Medical Association (AMA) and National Institutes of Health (NIH) recommend that patient education materials be written at no higher than a sixth-grade reading level. However, previous studies on the topic show that most patient materials on CTS exceed both the readability limits recommended by the AMA and NIH and the average reading ability of most US adults. ¹⁰ There is concern that with the large synthesis of online data that occurs when large language models generate responses, the responses may be more complex and more difficult to comprehend. Recent studies examined results when ChatGPT was asked the 10 most common patient questions regarding total hip replacement. They found that most answers required only minimal to moderate clarification (statements were correct and identified the finer points from established literature.)^11,12

The primary purpose of our study was to compare the responses of ChatGPT and Google Web Search regarding the FAQs regarding carpal tunnel surgery and examine their readability using defined indices. We hypothesized that the FAQs between the two programs would be comparable, but that ChatGPT would provide patients with more detailed responses for the FAQs than Google. In addition, we hypothesized that the responses from ChatGPT would have lower readability scores and be at higher reading levels.

Materials and Methods

A Google Web Search (www.google.com) was performed using a newly installed Google Chrome browser. Before the search, the browser was confirmed to be free of cookies and metadata from previous searches to minimize the effects of search algorithms. The term entered was “carpal tunnel release surgery.” The questions were, first, categorized into question topics: fact, policy, and value using the Rothwell’s classification system.^13,14 The Web site sources were categorized into commercial, academic, medical practice, single-surgeon practice, government, and social media using previously established categories from the literature (Table 1).^15,16 The 10 FAQs were obtained from the “People Also Ask” section. The inclusion criteria included any question with the term “carpal tunnel,” “carpal tunnel surgery,” and “carpal tunnel release.” Exclusion criteria included any repeated questions or questions unrelated to carpal tunnel surgery. The first 10 FAQs that met inclusion criteria were recorded (Table 2).

Table 1.

Rothwell’s Classification of Topic and Source Description.

Question topic	Fact	Objective information: specific activities, timelines, restrictions, technical details, cost
	Policy	Whether a specific course of action should be undertaken to solve a problem: indications, management, risks, complications
	Value	Evaluation of the desirability of an object, idea, event, or person: pain, longevity, evaluation of surgery
Web site source	Commercial	Organizations that provide public health information
	Academic	Universities, academic, medical groups, or academic societies
	Medical practice	Local hospitals or medical groups that have no academic affiliation
	Single-surgeon practice	Personal Web sites of individual surgeons
	Government	Web sites maintained by the national government
	Social media	Blogs, forums, online groups, nonmedical organizations

Note. Question topics by Rothwell’s classification: topics were categorized into 3 categories (fact, policy, and value) based on content. Web site source descriptions: categorized based on organizational ownership of the site.

Table 2.

Ten Frequently Asked Questions According to ChatGPT and Google.

ChatGPT	Google
What is carpal tunnel release surgery? (Fact)	How long is recovery from carpal tunnel surgery? (Fact)
How long does carpal tunnel release surgery take? (Fact)	How painful is carpal tunnel surgery? (Value)
Is carpal tunnel release surgery painful? (Value)	Is carpal tunnel release a major surgery? (Value)
What are the risks and complications of carpal tunnel release surgery? (Policy)	How long does carpal tunnel release surgery take? (Fact)
How long is the recovery period after carpal tunnel release surgery? (Fact)	What to expect immediately after carpal tunnel surgery? (Fact)
Will I need physical therapy after carpal tunnel release surgery? (Policy)	When is it too late for carpal tunnel surgery? (Policy)
How effective is carpal tunnel release surgery in relieving symptoms? (Fact)	What is worse after carpal tunnel surgery? (Policy)
Can carpal tunnel syndrome come back after surgery? (Value)	What are the cons of carpal tunnel surgery? (Policy)
Are there any alternative treatments to carpal tunnel release surgery? (Policy)	Why does carpal tunnel surgery hurt so bad? (Value)
What is the cost of carpal tunnel release surgery? (Fact)	Should I wear a wrist brace after carpal tunnel surgery? (Policy)

Note. Overlapping frequently asked questions between ChatGPT and Google are represented by bolded text. Question topic from Rothwell’s classification found in the parentheses.

ChatGPT-3.5 architecture was used with a capability of 175 billion parameters. The first inputted text in ChatGPT was “Record the ten most popular questions related to Carpal Tunnel Release Surgery.” The top 10 questions were recorded and compared with those from Google. Similar questions are represented in bold font (Table 2).

To properly compare answer content, ChatGPT was then asked to individually answer each of the 10 Google FAQs and provide sources.

Response length based on the number of words was determined and mean results were obtained for each program. Three different scoring systems were used to assess readability: Flesch Reading Ease score, Gunning-Fog Index, and Dale-Chall (Table 3).^{17 -19} Flesch Reading Ease score and Flesch-Kincaid score use the average length of sentences and the average number of syllables per word, the Gunning-Fog Index uses the count of sentences, words, and complex words consisting of 3 or more syllables in the text, and the Dale-Chall score is based off the Flesch-Kincaid score but uses a list of 3000 words that groups of fourth-grade American students could reliably understand, considering any word not on that list to be difficult.

Table 3.

Readability Analysis Score Descriptions.

Flesch Reading Ease score	100-90	Very easy to read, easily understood by an average 11-y-old student
	90-80	Easy to read
	80-70	Fairly easy to read
	70-60	Easily understood by 13- to 15-y-old students
	60-50	Fairly difficult to read
	50-30	Difficult to read, best understood by college graduates
	30-0	Very difficult to read, best understood by university graduates
Gunning-Fog grade level	6-8	Sixth grade to eighth grade
	9-12	High school freshman—senior
	13-15	College freshman—junior
	16	College senior
	17-20	Postgraduate
	20+	Postgraduate plus
Dale-Chall readability formula	<4.9	Easily understood by an average fourth-grade student or lower
	5.0-5.9	Easily understood by an average fifth- or sixth-grade student
	6.0-6.9	Easily understood by an average seventh- or eighth-grade student
	7.0-7.9	Easily understood by an average 9th- or 10th-grade student
	8.0-8.9	Easily understood by an average 11th- or 12th-grade student
	9.0-9.9	Easily understood by an average college student

Note. Readability analysis systems of Flesch Reading Ease score (high number represents text that is easy to read and low number is representative of more difficult text), Gunning-Fog index (score correlated to American grade level), Dale-Chall readability formula (score ranges 1-10 with grade level <4.9 indicating fourth grade level and 10 indicating college student grade level).

We used a qualitative analysis to compare the responses with numerical fact-based answers.

Independent samples t tests were used to compare the answer lengths based on the number of words as well as readability analyses. The significance level was set at P less than .05.

Results

Most FAQs on Google Versus ChatGPT

Of the 10 most FAQs on Google and ChatGPT, 40% were the same between the two search engines. The Google FAQ topics were 40% fact, 30% value, and 30% policy, whereas the ChatGPT topics were 40% policy, 30% value, and 30% fact (Table 2).

Characteristics of Sources Used in FAQ Responses

Google’s answers to FAQs were from one source per question: commercial (40%), academic (40%), government (10%), and single-surgeon practice (10%) Web sites. With ChatGPT, each answer was constructed from two different sources per question and the sources were all from academic Web sites including the American Academy of Orthopaedic Surgeons, American Society for Surgery of the Hand, Mayo Clinic, and Cleveland Clinic.

Comparison of Responses

A comparison between the answer content revealed that the mean response length of ChatGPT answers (289 ± 80 words) was longer than Google’s answers (43 ± 7 words; P < .001).

The Flesch Reading Ease score algorithm calculated Google to be at 46 ± 18 and ChatGPT at 29 ± 8 (P = .02). The Gunning-Fog index showed Google’s grade level to be 15 ± 4 and ChatGPT to be at 18 ± 2 (P = .02). The Dale-Chall’s readability formula determined Google’s readability grade level to be 9 ± 3 and ChatGPT’s to be 11 ± 1 (P = .03).

Qualitative analysis of the 3 questions that had numerical answers [(1) How long is recovery from carpal tunnel surgery? (2) How long does carpal tunnel release surgery take? (3) When is it too late for carpal tunnel surgery?] revealed similar answers (Table 4).

Table 4.

Google and ChatGPT Fact-Based Questions With Numerical Answers.

1. How long is recovery from carpal tunnel surgery?	Google	The recovery from carpal tunnel surgery takes time—anywhere from several weeks to several months. If the nerve has been compressed for a long period of time, recovery may take even longer. Recovery involves splinting your wrist and getting physical therapy to strengthen and heal the wrist and hand.
	ChatGPT	The recovery period from carpal tunnel surgery can vary depending on several factors, including the individual’s overall health, the specific surgical technique used, and how well postoperative instructions are followed. Although individual experiences may differ, here is a general timeline of the recovery process: 1. Immediate postoperative period: After the surgery, you will likely have a bandage or dressing on your hand, which should be kept clean and dry. Your hand may be elevated to help reduce swelling. You may also be prescribed pain medication to manage any discomfort. 2. Initial healing: In the first week or two, you may experience some swelling, bruising, and mild to moderate pain in the hand and wrist area. It is important to follow your surgeon’s instructions regarding wound care, activity restrictions, and medication usage. 3. Stitches removal: If nonabsorbable stitches were used, they are typically removed within 1 to 2 wk after the surgery. Your surgeon will advise you on when and where to have the stitches removed. 4. Hand therapy and rehabilitation: Your surgeon may recommend hand therapy or physical therapy to aid in your recovery. This may involve exercises, gentle stretching, and techniques to improve hand strength and mobility. The duration and frequency of therapy will depend on your individual needs. 5. Return to normal activities: Most individuals can resume light activities within a few weeks after surgery. However, it may take several weeks to a few months to fully recover and return to more strenuous activities or work that involves repetitive hand motions. It’s important to note that the timeline provided is a general guideline, and the actual recovery period can vary. It’s crucial to follow your surgeon’s postoperative instructions, attend follow-up appointments, and communicate any concerns or setbacks during the recovery process. Your surgeon will provide you with specific instructions tailored to your case, and they will be able to give you a more accurate estimate of the expected recovery duration based on your individual circumstances.
2. How long does carpal tunnel release surgery take?	Google	How long does carpal tunnel release surgery take? The surgery itself typically takes about 15 min. However, patients usually spend about 45 min in the operating room while equipment is set up and anesthesia administered.
	ChatGPT	The duration of carpal tunnel release surgery can vary depending on factors such as the surgeon’s technique, the complexity of the individual case, and whether it is an open or endoscopic procedure. However, on average, carpal tunnel release surgery typically takes around 15 to 30 min to perform. Open carpal tunnel release surgery, which involves making a small incision in the palm of the hand, is generally quicker and may take around 15 to 20 min. In this procedure, the surgeon cuts the transverse carpal ligament to relieve pressure on the median nerve. Endoscopic carpal tunnel release surgery, which is a minimally invasive technique using a small camera (endoscope) and specialized instruments, may take slightly longer, averaging around 20 to 30 min. This technique involves making 1 or 2 small incisions and visualizing the internal structures with the endoscope to guide the release of the ligament. It’s best to consult with your surgeon or health care provider for a more accurate estimate of the expected duration of the surgery based on your specific case. They can provide you with more detailed information and address any concerns you may have.
3. When is it too late for carpal tunnel surgery?	Google	Although it can be a bit more difficult to treat carpal tunnel if you’ve had symptoms for some time, it’s usually never too late for you to seek help so that you can comfortably write and use your hands while taking part in daily activities.
	ChatGPT	There is no specific “too late” point for carpal tunnel surgery. Carpal tunnel release surgery is typically recommended when conservative treatments have been ineffective in managing the symptoms of carpal tunnel syndrome or when the condition has progressed to a point where it significantly affects your daily activities and quality of life. It’s generally advisable to seek medical attention and consider surgery earlier rather than later to prevent further nerve damage and maximize the chances of successful outcomes. However, the decision to undergo carpal tunnel surgery ultimately depends on individual factors and the advice of a health care professional or surgeon. In some cases, individuals may delay seeking medical intervention for various reasons, allowing the condition to progress. Although the severity of the symptoms and the extent of nerve damage can affect the prognosis and recovery, carpal tunnel release surgery can still be considered even in advanced stages of the condition. If you suspect you have carpal tunnel syndrome or if you are experiencing symptoms such as pain, numbness, tingling, or weakness in your hand or wrist, it is advisable to consult with a health care professional. They can evaluate your condition, discuss treatment options, and determine if carpal tunnel release surgery is appropriate for you based on your individual circumstances.

Note. Fact-based question that had numerical answers in the left column and answers from ChatGPT and Google in the right column with numerical answers are in bolded text.

Finally, all ChatGPT answers ended with some variation of an instruction to consult your surgeon or health care provider to get patient-specific and more accurate answers. Every answer had a modification of the statement, “Every individual patient is different and may require care instructions specifically tailored to their case.”

Discussion

ChatGPT can serve as a valuable educational resource for patients looking to understand their hand surgery pathologies and procedures. Our findings reveal that there was a substantial overlap in the most common carpal tunnel–related questions between ChatGPT and Google; that ChatGPT and Google had similar ratios of fact-, value-, and policy-based questions; and that ChatGPT provided longer, more detailed responses than Google. In the provided answers, ChatGPT used two academic sources, whereas Google used only one source, which was typically either commercial or academic. Despite using different sources, the 3 questions that had numerical-based answers revealed comparable outputs. On readability analysis, Google’s responses were easier to read and were at a lower grade level than ChatGPT’s.

The rise of the internet, Google, and other search engines has altered the physician-patient interaction. With access to a wealth of information online, patients have become able to research their symptoms and treatments before and after appointments with physicians. Although there are many benefits to having well-informed patients, clinicians may also have to address the preexisting framework and biases that patients have from such antecedent research. ²⁰ Although the increase in access to information allows patients to become more educated about their condition and has been shown to improve patient trust in their physician, it has also increased patient anxiety.^21,22 With the emergence of ChatGPT, one must question if this new technology (also known as Doctor ChatGPT) will replace or augment “Doctor Google.” The results of our study may indicate that despite being more evidence based, responses from ChatGPT may be more challenging to understand and therefore not as practical as a patient education tool.

Despite our findings, ChatGPT has many benefits over Google. First, it generates human-like text that is synthesized from multiple sources drawing from a large corpus of data. Whereas Google is often limited by showing a single source for a question or broad topic, ChatGPT can perform numerous language completion tasks. ChatGPT is able to take context into account in its answers and thus can allow a user to tailor a response to a specific context or need. For example, when asking ChatGPT a question about carpal tunnel surgery, a patient could state they would like the answer to be written at a specific grade level so as to suit their level of education and understanding. Both Google and ChatGPT have limitations and biases that can undermine their utility and accuracy of response. In allowing sponsored results, Google shows the user sources that may not be best suited or the most accurate source available, which is reflected in our finding that commercial sources are among the most common from Google. Other factors such as location, search history, and settings contribute to Google’s algorithm of what to show a user, making results quite subjective to the user, rather than directly related to the query itself. In contrast, although ChatGPT eliminates the biases of tailoring results to user settings, location, and other factors, bias still exists in that ChatGPT’s answers will be limited to its training data.

Since its entry into the market, there has been investigation into the utility of ChatGPT in all aspects of health care, including its role of patient-facing online content, analogous to the role of “Doctor Google.”^23,24 Several studies have assessed ChatGPT’s responses to common patient questions or scenarios in the fields of obstetrics/gynecology, otolaryngology, plastic surgery, and orthopedic surgery.^{21,22,25 -27} These studies have found mixed results in the accuracy and utility of ChatGPT. One found that when comparing ChatGPT and Google, ChatGPT performed better than Google for general medical advice, but worse when disseminating medical recommendations. ²⁵ Others emphasized the ease of readability in ChatGPT’s human-like responses but underscored the concern of this tool disseminating inaccurate, incomplete, and/or impersonalized information. ²⁶ In assessing the concern of inaccurate responses by ChatGPT, another study found that only 7% of ChatGPT responses had both accurate and authentic sources, highlighting an alarmingly high amount of inaccurate or incorrectly sourced answers provided by ChatGPT. ²⁸ This emphasizes the need for both patients and clinicians to use ChatGPT with caution and warns physicians that the information their patients may have consumed on ChatGPT may have been inaccurate, incomplete, or incorrectly sourced.

This study is not without limitations. Only a small sample of 10 questions were compared. Although we did our best to control for biases in Google’s answers based on user characteristics, there is the possibility we did not completely eliminate this with responses being influenced by unforeseen factors such as the “location services” on the computer being turned on.

Overall, ChatGPT can provide evidence-based responses from academic Web sites on the topic of carpal tunnel release surgery and the issues with its poorer readability may be modifiable if users ask for more readable responses, a feature not available in Google. Future studies should assess patient and surgeon perspectives on the role of ChatGPT in answering hand surgery–related questions, the accuracy of ChatGPT’s responses, and how the use of ChatGPT may alter patient behaviors of health care utilization.