ChatGPT Is Not Yet Ready to Replace Motility Experts

Large language models (LLMs) are rapidly emerging as transformative tools in patient care and clinical practice. Among these, ChatGPT (Chat Generative Pre-Trained Transformer) has achieved widespread popularity, with over 200 million active weekly users.¹ The application of artificial intelligence (AI) to high-resolution esophageal manometry (HRM) is well suited as it can efficiently analyze large volumes of data and images with distinct, stereotypical patterns.² Despite this potential, the application of ChatGPT to HRM has not been previously studied. We sought to evaluate the performance of ChatGPT-4o, an enhanced version capable of processing combined text and image inputs, in classifying HRM data according to the Chicago Classification v4.0 (CCv4.0) schema.³

HRM studies were selected from a prospectively enrolled cohort for an NIH-funded study (P01 DK117824) of adult patients (ages 18–89) evaluated for dysphagia at the Esophageal Center of Northwestern Memorial Hospital between 2019 and 2022. HRM studies were stratified by CCv4.0 diagnosis and ten studies were selected at random from each diagnostic group. Exceptions were for distal esophageal spasm (DES) and hypercontractile esophagus (HE), where fewer patients necessitated the inclusion of all available cases. Patients with a history of foregut surgery, mechanical obstruction, or conditions such as esophageal stricture, eosinophilic esophagitis, severe reflux esophagitis (Los Angeles classification C or D), or hiatal hernia >3 cm were excluded. Written informed consent was obtained from all participants for maintenance of data. All identifying information was removed, and only topographic plots and HRM-metrics (e.g., integrated relaxation pressure [IRP], panesophageal pressurization, distal contractile integral) were applied for analysis of this minimal-risk study under a waiver of informed consent (study protocol was approved by Northwestern University IRB).

Performance of ChatGPT-4o for identifying CCv4.0 diagnoses was evaluated across five distinct prompts. Prompt 1 provided a single image containing 10 supine swallows, 5 upright swallows, and the rapid drink challenge without metrics. Prompt 2 added thumbnails illustrating representative topographic patterns for each CCv4.0 diagnosis. Prompt 3 further included mean IRP values, study references, and panesophageal pressurization details. Prompt 4 entailed standard HRM reports with summary metrics for individual swallows, excluding CCv4.0 diagnoses. Finally, Prompt 5 provided a spreadsheet summarizing all key metrics while omitting diagnoses; Table S1. To mitigate bias from prior inputs, the chat session was reset before each HRM plot was submitted

Performance metrics included accuracy and Cohen’s kappa, calculated for each prompt and stratified by CCv4.0 diagnosis. Confusion matrices were constructed to elucidate error patterns. Analyses were performed using R version 4.4.0.

A total of 85 HRM studies were identified, with 10 representative studies for each CCv4.0 diagnosis, except for DES (6 studies) and HE (9 studies). Results revealed consistently poor performance across all prompts, with incremental improvement as additional structured information was provided. Accuracy and kappa values for Prompts 1 through 5 were 0.09/0.20, 0.07/0.18, 0.11/0.21, 0.33/0.41, and 0.35/0.41, respectively; Figure 1. Confusion matrices revealed a strong bias toward type II achalasia and ineffective esophageal motility (IEM), particularly for Prompt 3; Figure S1.

Figure 1 – ChatGPT Accuracy and Kappa Statistic by Prompt.

1A: Accuracy of ChatGPT-4o in classifying high-resolution esophageal manometry data across five prompts of increasing complexity. 1B: Cohen’s kappa measuring agreement between ChatGPT-4o classifications and high-resolution esophageal manometry (CCv4.0) diagnoses.

Both metrics show an upward trend, with accuracy increasing from 0.09 for Prompt 1 to 0.35 for Prompt 5, and kappa values rising from 0.20 to 0.41. Agreement levels based on kappa values: 0–0.2 (poor), 0.2–0.4 (fair), 0.4–0.6 (moderate), 0.6–0.8 (good), and 0.8–1.0 (excellent).

This bias likely stems from data that type II achalasia and IEM are the most common CCv4.0 diagnoses among disorders of esophagogastric junction (EGJ) outflow and peristalsis respectively.^4,5 Despite this, the model struggled to accurately identify “normal” motility, the most common manometric pattern, likely due to its underrepresentation in the medical literature.

Providing more information, as seen in the progression from Prompts 1 to 5, numerically improved performance. Text-based prompts consistently outperformed image-based ones, highlighting ChatGPT’s limitations in interpreting medical images. Even with complete metrics, the model failed to reliably apply the CCv4.0 scheme. Performance was further undermined by a tendency to overestimate IRP values and significant response instability, with identical prompts often producing inconsistent classifications.

This study’s findings align with prior reports of LLM limitations in medical image analysis. One recent study assessed the accuracy of ChatGPT-4 in identifying colorectal adenomas from histopathologic images revealing a sensitivity of 74%, but poor specificity of 36% and low diagnostic consistency, with kappa values ranging from 0.06 to 0.11.⁶ Similarly, in another study evaluating Chat-GPT’s performance in diagnosing liver diseases from computed tomography images, sensitivity was found to be 20% and specificity 30%.⁷ Although the addition of textual context to images has been shown to improve ChatGPT’s accuracy, overall diagnostic performance remains suboptimal.⁸ These limitations are not surprising given that ChatGPT is optimized for language processing. AI platforms designed specifically for medical image classification, equipped with domain-specific neural network architectures, may yield superior results. LLMs like ChatGPT are trained on large, unstructured datasets that include conflicting and sometimes inaccurate information, reflecting the inconsistencies of “world knowledge” rather than the rigor of domain-specific guidelines such as CCv4.0.⁹ Consequently, these models are prone to generating factually inconsistent “hallucinations”.

While the findings from this study highlight the significant limitations of ChatGPT for HRM interpretation, they should not detract from the promising advancements being made with dedicated AI systems in this field. Models have been developed using specialized algorithms trained on specific, high-quality HRM datasets with architectures optimized for pattern recognition in manometric data.¹⁰ These purpose-built systems have demonstrated considerable promise and represent the appropriate direction for AI implementation in esophageal motility diagnostics. Future research should continue to refine these specialized approaches rather than attempting to repurpose general-purpose LLMs for clinical diagnoses.

These findings underscore the necessity for caution when considering LLMs for clinical applications. Until more refined and dedicated systems are developed, reliance on ChatGPT for HRM interpretation should be avoided, as it risks introducing significant errors into patient care. This serves as a cautionary tale for esophagologists seeking to leverage AI to streamline their workload. While LLMs hold tremendous potential, their current limitations preclude their use as standalone diagnostic tools in esophageal motility disorders.