Editor’s Spotlight/Take 5: Artificial Intelligence Shows Limited Success in Improving Readability Levels of Spanish-language Orthopaedic Patient Education Materials

Here at Clinical Orthopaedics and Related Research®, the best use cases we’ve seen for large language models powered by artificial intelligence—tools like ChatGPT, Gemini, and Claude—have involved helping people to write more clearly. Whether helping authors translate their manuscripts into English from other languages [3] or presenting research in ways that are more accessible to nonexperts [4], we’ve been open to authors using AI in their papers as long as they disclose the use of these tools. Anecdotally, the writing generated by these tools often seems a little “off” to me—adjectives and adverbs are placed where one might not expect or want them, for example. But when compared with prose that is written by people who struggle to compose in English, the AI-aided result usually is pretty serviceable.

For that reason, when I read in this month’s Editor’s Spotlight/Take 5 article [1] that the most-recent version of ChatGPT (GPT-4) was not effective at adapting Spanish-language orthopaedic educational materials to the average United States reading level (5^th grade), I was surprised, and more than a little disappointed. After all, we at CORR® are not the only ones who think these tools should make us more effective in these ways; according to one industry-leading company, the top five use cases for these tools involve content creation and writing assistance, language translation, and healthcare applications [5]. Yet, a multicenter group led by Roland A. Navarro MD of the Southern California Permanente Medical Group in Harbor City, CA, USA (Fig. 1), found that ChatGPT-4 adapted Spanish-language educational materials from the American Academy of Orthopaedic Surgeons (AAOS) OrthoInfo website to the 5^th grade level or lower in only 39% of attempts [1]. Worse still, only about half of the AI-created adaptations were classified as accurate or useful by expert readers.

Fig. 1.

Ronald A. Navarro MD

After doing this for 12 years, I’m not often surprised, but this paper’s main findings [1] surprised me. I’ve used these tools for related purposes and watched authors use them, and my Bayesian priors all involved the belief that ChatGPT and other large language models are fit to task for this kind of work. In fact, we’re about to roll out plain language summaries for CORR’s website, and I’ve been pleased with the beta version of that process. In all the examples it has generated for us, clarity and accuracy have seemed more than fine. But now that I’ve learned of Dr. Navarro’s discoveries, I’ll be watching it like a hawk. A very nervous hawk.

The writing generated by large language models like ChatGPT doesn’t read like Hemingway. Then again, what I write doesn’t, either. It’s a problem, though, if the medical output from these tools (or from the average journal editor) is unhelpful, inaccurate, or misleading. So join me in the Take 5 interview that follows with Dr. Navarro, senior author of “Artificial Intelligence Shows Limited Success in Improving Readability Levels of Spanish-language Orthopaedic Patient Education Materials” [1], to get a sense for why ChatGPT fell short, what we should do in the meantime, and when we might expect better results from these tools.

Take 5 Interview with Ronald A. Navarro MD, senior author of “Artificial Intelligence Shows Limited Success in Improving Readability Levels of Spanish-language Orthopaedic Patient Education Materials”

Seth S. Leopold MD: Congratulations on this somewhat dispiriting but eye-opening paper. Characterize your impressions in a few main buckets: To what degree were the adaptations ChatGPT created for you unhelpful, inaccurate, or misleading? Can you share some examples?

Ronald A. Navarro MD: In several cases, the outputs generated by ChatGPT introduced confusion or strayed from the original meaning, which limited their effectiveness. This resulted in some adaptations being unhelpful, inaccurate, and, at times, misleading for Spanish-language readers (Fig. 2). Adaptations were considered unhelpful when they did not clearly explain a procedure or when they relied on complex technical terms instead of analogies and plain language for the explanations they did provide. Misleading information tended to exaggerate or oversimplify things—for example, by labeling materials as “toxic” when they are not or by suggesting conditions are “always” caused by something when they are not—whereas inaccurate information was completely incorrect.

Fig. 2.

Examples of how ChatGPT-4 adaptations of OrthoInfo educational materials can be unhelpful, inaccurate, or misleading to Spanish-speaking patients.

Dr. Leopold: In the demo version of the tool we’re using that creates plain language summaries of abstracts in CORR, my impression is that they’re pretty good. Not down to 5^th grade level, by any means, but more comprehensible than the originals. And, importantly, they seem accurate. Is there something different about adapting in a language other than English? Is it that a 5^th grade reading level is just too hard to get to for medical content? Something else?

Dr. Navarro: My opinion is that using AI to simplify texts in languages other than English to a 5^th grade reading level will likely lead to a loss of essential meaning that renders the final content inaccurate and useless for patient education. One key issue with adapting content in languages other than English is the complexity of the translation thanks to the nuances of medical terminology. The AI tool’s ability to accurately translate and simplify technical terms into understandable language while preserving medical precision remains limited. Additionally, in Spanish, achieving clarity often requires different sentence structures or phrases that may not have an equivalent in English. It’s important to recognize that a balance must be struck between simplifying the language for easier understanding and maintaining the accurate and necessary information that is required for effective patient education. It’s not necessarily that the 5^th grade level is unattainable, but achieving it in a way that does not oversimplify or distort critical information is where the challenge lies, particularly in languages like Spanish (Fig. 3).

Fig. 3.

Example of ChatGPT-4 failing to adapt a patient education material to a 5th grade reading level.

Dr. Leopold: You’ve identified a problem and found that a promising-looking solution fell short. What should readers do in the meantime if they want to disseminate patient information materials that those patients can understand and use?

Dr. Navarro: I think it’s fair to query the patient on their education level while simultaneously directing them to resources like the Spanish language materials at orthoinfo.aaos.org. Then, it would be helpful to ask patients if they have any questions or areas that lack clear understanding. Hopefully, the clinician has access to interpreters in person or remotely who can help the patient shore up areas that are unclear. The key takeaway is that while the OrthoInfo site is a valuable resource, it should be considered a supplement to, not a substitute for, comprehensive patient education. We recognize that from a physician’s perspective, educating a Spanish-speaking patient may take more time because of the need for interpreters. This may lead some physicians to simply provide the online resources and move on. However, to proactively improve outcomes for this underserved patient population, we must ensure they receive the same level of patient education as English-speaking patients.

Dr. Leopold: To what degree might other large language models be more suitable to this task than ChatGPT? While ChatGPT is perhaps the most widely known, there are dozens of others [2]…

Dr. Navarro: I actually asked ChatGPT-4 a series of questions on February 27, 2025, and you’re right; there are many options. I started off asking, “Which large language model is best at creating patient information and translating patient information from English into Spanish and other languages?” Of course, in a nonsentient manner, ChatGPT listed itself as a great resource, along with MedPaLM (Google), BioGPT (Microsoft), and No Language Left Behind (NLLB, Meta), as all have a biomedical focus and are effective at translation. DeepL was also mentioned for its translation capabilities, although it is technically not an LLM. Interestingly, aside from ChatGPT, none of those were listed in the reference you cited [2].

For adapting patient information to a desired reading level, I asked, “For the above query, what LLMs are best at translating to a 5th grade level while retaining essential meaning?” ChatGPT-4, MedPaLM, or BioGPT were recommended. For translations, the suggested approach was to use ChatGPT-4 or MedPaLM paired with DeepL or NLLB. Finally, when I asked which tools would provide translations at a 5^th grade reading level that would best preserve meaning, the suggestions included GPT-4, MedPaLM, NLLB, DeepL, and a new addition—Claude (Anthropic)—which was referenced in Kerner [2].

Overall, ChatGPT-4, MedPaLM, and BioGPT seem to be the best choices for adapting patient information, while DeepL and NLLB appear to be best for translation accuracy. At this moment in time, the best strategy is selecting the most effective tool based on the specific need, which will be a challenge for busy clinicians.

Dr. Leopold: Do you have a sense for the pace of development of these tools? When might your paper be rendered out of date because the performance of these large language models is sufficient to solve the problem you’ve set out to solve?

Dr. Navarro: While I am nowhere near as versed on these issues as the likes of Elon Musk, Sam Altman, and others, I believe that the speed of change, improvement, and adoption of newer, more specialized medical patient information translation tools will render current tools obsolete within the next 5 years—at least insofar as better alternatives emerge. The real challenge lies in ensuring that the average physician or patient can easily access the best available tool for patient-friendly, grade level–appropriate medical information. If there are so many tools, and the average person isn't aware of all of them (as likely is the case), then there will be a fragmentation of use. In the way that Apple somewhat standardized the smartphone, could there be similar standards and regulations for ensuring patient comprehension of medical information in multiple languages? Who would lead this effort, and would it be a top-down initiative? I'm not sure. Ultimately, I think the question of improving communication with patients extends beyond providing patient-friendly information on their conditions—the scope includes patient comprehension of care pathways, informed consent, and one’s agency (as an active participant in decisions regarding their own health). That is an even greater challenge, and we hope this work will foster continued improvement in patient communication across multiple languages.