Completeness and readability of GPT-4-generated multilingual discharge instructions in the pediatric emergency department

Abstract

Objectives

The aim of this study was to assess the completeness and readability of generative pre-trained transformer-4 (GPT-4)-generated discharge instructions at prespecified reading levels for common pediatric emergency room complaints.

Materials and Methods

The outputs for 6 discharge scenarios stratified by reading level (fifth or eighth grade) and language (English, Spanish) were generated fivefold using GPT-4. Specifically, 120 discharge instructions were produced and analyzed (6 scenarios: 60 in English, 60 in Spanish; 60 at a fifth-grade reading level, 60 at an eighth-grade reading level) and compared for completeness and readability (between language, between reading level, and stratified by group and reading level). Completeness was defined as the proportion of literature-derived key points included in discharge instructions. Readability was quantified using Flesch-Kincaid (English) and Fernandez-Huerta (Spanish) readability scores.

Results

English-language GPT-generated discharge instructions contained a significantly higher proportion of must-include discharge instructions than those in Spanish (English: mean (standard error of the mean) = 62% (3%), Spanish: 53% (3%), P = .02). In the fifth-grade and eighth-grade level conditions, there was no significant difference between English and Spanish outputs in completeness. Readability did not differ across languages.

Discussion

GPT-4 produced readable discharge instructions in English and Spanish while modulating document reading level. Discharge instructions in English tended to have higher completeness than those in Spanish.

Conclusion

Future research in prompt engineering and GPT-4 performance, both generally and in multiple languages, is needed to reduce potential for health disparities by language and reading level.

Background and significance

Since the COVID-19 pandemic, emergency departments have been struggling with increased patient volumes and staffing shortages.¹ Prewritten discharge instructions are sometimes recommended to increase emergency department efficiency;² however, discharge instructions more generally are often above patients’ health literacy levels³ or not in their preferred language at all.^4,5 In prior studies, over 90% of clinicians indicated that generating language-concordant discharge instructions represented a barrier to care, and 26% of children’s hospitals do not translate discharge instructions.^6,7 Additionally, even when language-concordant discharge instructions are provided to patients, Spanish-language instructions are often much less complete than English-language ones.⁴

This language discordance becomes an issue when one considers that people with limited English proficiency (LEP) experience worse healthcare quality than their English-proficient counterparts,^8,9 especially since Spanish-speaking parents of pediatric patients have indicated they would understand discharge instructions better if information was written in Spanish and that they would prefer to receive instructions in Spanish.¹⁰ Patient-provider language discordance during discharge has also been associated with reduced understanding of medication category and purpose¹¹ as well as increased rates of readmission.^12,13

The reasons for the use of English discharge instructions for Spanish-speaking patients are many, including timing issues in coordination of document translation and patient discharge as well as shortages in staffing and funding for document translation.⁶ In fact, these issues have led some clinicians to use software such as Google Translate to generate discharge instructions, even though this practice can generate instructions that might lead to harm in more complex cases.^14,15 Further innovation in the process of producing multilingual discharge instructions is therefore needed to increase the efficiency of emergency departments and ameliorate the health inequities that current practices can cause.

Generative large language models (ie, artificial intelligence tools designed to generate text based on user-provided inputs) such as generative pre-trained transformer-4 (GPT-4) may serve as useful tools to help busy clinicians generate discharge instructions for emergency department patients. GPT-4 has shown promise in writing intelligible and coherent discharge summaries and instructions for patients post-surgery,^16,17 modulating the medical complexity of texts,¹⁸ and in answering patient medical questions in Spanish.¹⁹ Therefore, inputting a query requesting discharge instructions in Spanish into a large language model may be an efficient alternative to writing and then translating discharge instructions.

For GPT-4-generated discharge instructions to be useful, they must contain the necessary information to guide care at home and prompt re-presentation to the hospital when necessary. Furthermore, discharge instructions must be at a reading level that patients can understand. To our knowledge, no literature currently exists examining the specific issues of completeness and readability of GPT-4-generated discharge instructions in English and in Spanish. Therefore, the current study’s primary objective is to systematically evaluate multilingual GPT-4-generated discharge instructions in terms of completeness and readability in both English and Spanish. A comparison of discharge instructions’ completeness and readability by language will help to both evaluate large language models’ current potential for the generation of discharge instructions as well as identify language-specific shortcomings that may disproportionally affect Spanish-speaking patients as these models continue to be adapted for use in healthcare settings.

Materials and methods

Generation of GPT-4 output corpus

In June of 2023, GPT-4 was used to create discharge instructions at fifth- and eighth-grade reading levels for 6 common pediatric ED complaints (diarrhea and vomiting, bronchiolitis, fever, abdominal pain, asthma, and minor head injury without concussion), which constitute the chief complaint for approximately two-thirds of pediatric emergency department visits.²⁰ GPT-4 was chosen over GPT-3.5 due to prior evidence suggesting GPT-4’s superior handling of non-English languages.²¹ Before analysis was carried out, the input prompt was iterated and refined to specify pediatric (versus general) diagnosis and discharge instructions for parents (versus simply discharge instructions). These changes were made after previous prompts generated discharge instructions for adult (versus pediatric) complaints or were directed to the pediatric patients (as opposed to their parents). The term emergency department was not used so as to allow for generalizability of results to those cases where patients are discharged soon after admission (eg, for intravenous rehydration in the setting of severe vomiting). In addition, while no specific guidance regarding length of discharge instructions exists, avoiding excessively lengthy text generally has been used in prior studies to ensure discharge instruction readability²²; furthermore, the authors wanted to avoid rewarding GPT-4 outputs that contained excessively verbose text. Responses were therefore capped at 500 words, a length that allowed for sufficient word space to produce complete discharge instructions while preventing excessively verbose outputs. The final prompt used was “Create discharge instructions (in under 500 words) for pediatric [diagnosis] at a[n] [5th/8th]-grade reading level in [Spanish/English] for parents.” Each instruction case was generated 5 times to appropriately capture the breadth of potential responses from GPT-4. A new session was used for the generation of each output. Therefore, given 6 chief complaints, each with 4 cases (2 languages and 2 grade levels), with each case being generated 5 times, 120 total outputs were generated.

Scoring of outputs

Discharge instructions were analyzed in terms of completeness and readability by 2 individuals, both of whom are clinical medical students and certified English-Spanish medical interpreters. Completeness was assessed based on a previous report of the most important elements of discharge instructions according to emergency room clinicians²³: given that no universally accepted criteria exist for the specific return criteria that must be included in pediatric discharge instructions,²⁴ the 5 most important elements of discharge instructions for each diagnosis were extracted from the results of a previous study of ED clinicians (see Table S1).²³ Briefly, this study used a modified Delphi method with 37 clinicians to converge on those discharge instruction features considered most important to include.

To measure completeness, the presence of each discharge instruction criterion in each output was coded as present or absent by the 2 raters. To assess inter-rater reliability, Cohen's kappa was examined across raters. Kappa values were deemed unsatisfactorily low, and so specific discordances between raters were identified (see online repository file completeness_Scores_Comparisons.xlsx), and the reason for discordance determined. Then, a third, nonclinical author of the paper was consulted to understand how these discordances would be interpreted by a nonclinical audience (eg, parents) and create final scoring rules for these discordances (see Resolving Discordances in the Supplementary Methods).

Quantification of completeness

Completeness for each instruction was calculated by measuring the proportion of most important elements contained within the generated output and was reported as a percentage. It was also noted that the top 5 items to include in discharge instructions were not unanimously agreed upon by all clinicians for all criteria in the cited Delphi study. While this lack of agreement is expected given normal variations in clinical judgment, this variability in the top 5 items to include could potentially impact the utility of raw completeness scores to determine GPT-4’s ability to generate complete discharge instructions. For example, if the agreement amongst experts for the top 5 return criteria for a given chief complaint were only 80%, the average completeness rating of a discharge instruction containing only those top 5 criteria (if rated by those same experts) would be only 80% (see Proof S1 in the Supplemental Methods for a demonstration of this). Therefore, we endeavored to create a measure comparing the raw completeness of the GPT-4 output to the average expected completeness rating of an “ideal” discharge summary (ie, one containing all top 5 criteria). This was calculated by calculating the ratio of the raw completeness score and the mean percent consensus amongst raters for each chief complaint (see Table S1 for these consensus scores). Completeness scores, therefore, can be used to gain an intuitive sense of GPT-4’s ability to incorporate essential discharge information into discharge instructions; completeness ratios, on the other hand, can help readers interpret these completeness scores in the context of normal variation in clinical opinion.

Quantification of readability

Previous research has demonstrated that approximately 80% of ED attendees have a reading level of eighth grade or lower,²⁵ although more recent research has shown an average reading ability between a seventh- and eighth-grade level in 1 urban ED.²⁶ Therefore, an eighth-grade reading level was chosen as one of the test cases. Additionally, the American Medical Association has previously recommended that patient education materials be written at a fifth- or sixth-grade reading level;²⁷ therefore, a fifth-grade case was also included. Readability was quantified using the Fernandez-Huerta readability score for Spanish or the Flesch-Kincaid readability score for English, both of which have previously been used to compare the readability of patient education texts in English and Spanish. For these goal reading levels, a reading score between 70 and 80 corresponds to a reading level of eighth or ninth grade in both these systems, while a score between 90 and 100 corresponds to a reading level of fifth grade (as a higher score indicates higher readability and therefore a lower corresponding grade level).²⁸

Statistical analysis

Prior to statistical analysis, both raters read all GPT-4-generated outputs to ensure that none contained instructions that would cause harm.

The predictor variables of interest were output language and reading level case; the outcome variables of interest were completeness, completeness ratio, and Flesch-Kincaid or Fernandez-Huerta reading level. To characterize the general performance of the GPT-4 for the task, the overall completeness, completeness ratio, and reading level of all outputs were calculated. Next, to determine GPT-4’s general performance in English versus Spanish, aggregated outputs (ie, all outputs in the fifth-grade and eighth-grade cases combined) in English and in Spanish were compared in terms of completeness (including completeness ratio) and readability. Then, to determine GPT-4’s general ability to modulate reading level without omitting important information, aggregated outputs (ie, all outputs in English and in Spanish combined) in the fifth-grade and eighth-grade reading level cases were compared in terms of completeness (including completeness ratio) and readability. Finally, to better understand how specific grade-level cases may contribute to language-specific issues with completeness or readability, completeness (including completeness ratio) and readability were compared across languages for outputs at a specific grade level (eg, for English versus Spanish outputs in the fifth-grade reading level case).

Differences between conditions’ completeness and reading levels were analyzed using independent 2-sample t-tests, with Levene’s test of equal variance used to determine appropriate t-value for analysis. All significances reported were 2-sided. Analyses were carried out using SPSS statistics version 29.0.1.0.

Results

General performance and characterization of outputs

Outputs were legible and comprehensible in English and in Spanish (Table 1) and did not contain any outputs giving potentially harmful advice. Kappa values of inter-rater reliability ranged from 0.05 to 0.68 (Table S2). The mean completeness of all outputs was 57 ± 21% (mean ± standard deviation), the mean completeness ratio was 74 ± 28%, and the mean readability score was 70.5 ± 6.6 (corresponding to a seventh-grade reading level). By chief complaint, the mean completeness was 56 ± 23% (mean ± standard deviation) for diarrhea and vomiting, 59 ± 26% for bronchiolitis, 62 ± 14% for fever, 59 ± 22% for abdominal pain, 56 ± 20% for asthma, and 52 ± 18% for minor head injury without concussion. For each case (ie, each set of 5 outputs with the same chief complaint, language, and grade level), the standard deviation of completeness had a range between 9% and 30%. (Please see the repository file completeness_Scores.xlsx for the standard deviations for specific cases.)

Table 1.

Sample outputs from the minor head injury without concussion prompt.

Grade level	English	Spanish
Fifth grade	Dear Parents or Guardians, Your child has had a minor head injury. Good news is, it’s not a concussion, and doctors believe it is OK for your child to go home. Still, it’s important to keep an eye on them and help them get better. Here’s what you should do: Rest and relaxation: Your child needs to rest a lot. They should avoid rough play, running, bike riding, or anything that could hurt their head again. Even activities that need a lot of thinking, like schoolwork, should be reduced for a few days. Pain relief: If your child feels some pain, you can give them an over-the-counter medicine like Tylenol (or its generic form, acetaminophen). Don’t give them aspirin—it’s not safe for kids. Remember to always follow the instructions on the bottle…	Instrucciones para el Alta: Lesión Menor en la Cabeza de su Hijo   Estimado(a) padre/madre o tutor(a): Vigile los Síntomas: Su hijo ha sufrido una lesión menor en la cabeza, pero no ha mostrado signos de conmoción cerebral. Aun así, vigile a su hijo por si aparecen nuevos síntomas, como dolor de cabeza fuerte, vómitos, mareos, dificultad para caminar o hablar, o cambios en su comportamiento. Dolor: Para el dolor, puede darle a su hijo medicamentos sin receta, como paracetamol (Tylenol). Siga las instrucciones de envase para la dosis adecuada. No le de aspirina a su hijo…
Eighth grade	Dear Parent/Guardian, Your child has had a minor head injury. We’re glad to tell you that there are no signs of a concussion, but we do need you to monitor them at home for any changes. Rest and activity   Ensure your child gets plenty of rest in the first 24-48 hours. This helps the healing process. Avoid vigorous physical activities, like running or playing sports, until they feel completely well again…	Instrucciones de Alta para un Lesión Menor de Cabeza Sin Conmoción en Pediatría   Estimados padres, Su hijo ha tenido una lesión menor en la cabeza pero no muestra signos de conmoción cerebral. A continuación, encontrará las instrucciones a seguir en casa para ayudarle a recuperarse. Cuidado en casa   Descanso. Asegúrese de que su hijo descanse durante las primeras 24-48 horas después del incidente. Supervisión: Preste atención a cualquier cambio en su comportamiento o estado de ánimo. Aunque no tenga conmoción cerebral, aún debe ser observado por posibles síntomas…

English versus Spanish outputs: completeness and readability

Overall, English discharge instructions at all reading levels (n = 60) were significantly more complete than those in Spanish (n = 60). Specifically, in each language, 6 discharge diagnoses were examined, with 5 outputs generated for each at goal reading levels of fifth- and eighth-grade (ie, the fifth-grade and eighth-grade cases, respectively). English discharge instructions had a mean completeness of 62 ± 3% (mean ± standard error of the mean), while those in Spanish had a completeness of 53 ± 3% (P = .02). Readability (averaged across all outputs) did not differ significantly between languages. Completeness ratios followed a similar pattern, as English discharge instructions had an average completeness ratio of 80 ± 4% (mean ± standard error of the mean), while those in Spanish had an average completeness ratio of 68 ± 3% (P = .02) (Table 2).

Table 2.

Completeness and readability in all English versus all Spanish outputs.

Output attribute	All English (n = 60)	All Spanish (n = 60)	Difference
Completeness, mean ± SEM	62 ± 3%	53 ± 3%	t(118) = 2.33, P = .02
Completeness ratio, mean ± SEM	80 ± 4%	68 ± 3%	t(188) = 2.38, P = .02
Readability, mean ± SEM	70.73 ± 1.08	70.27 ± 0.55	t(87) = 0.38, P = .70

Fifth-grade versus eighth-grade outputs: completeness and readability

No significant difference in completeness or completeness ratio was noted between aggregate (ie, both English and Spanish) outputs for the 2 grade levels. Readability scores were significantly higher in the fifth-grade case discharge instructions as an aggregate (mean = 74.4 [corresponding to seventh grade], SEM = 0.70, n = 60) compared to the eighth-grade case (mean = 66.6 [eighth to ninth grade], SEM = 0.69, n = 60); P < .001 (Table 3).

Table 3.

Completeness and readability in all fifth-grade versus all eighth-grade cases.

Output attribute	All fifth grade (n = 60)	All eighth grade (n = 60)	Difference
Completeness, mean ± SEM	55 ± 3%	59 ± 2%	t(118) = −1.06, P = .29
Completeness ratio, mean ± SEM	72 ± 4%	77 ± 3%	t(118) = −0.98, P = .33
Readability, mean ± SEM	74.4 ± 0.70	66.6 ± 0.69	t(118) = 7.91, P < .001

English versus Spanish outputs stratified by grade level case: completeness and readability

In the fifth-grade case, English and Spanish outputs did not differ significantly in completeness or completeness ratio; however, English outputs were significantly more readable than Spanish ones in the fifth-grade case (English: mean = 77.24 [seventh grade], SE = 0.98, n = 30; Spanish: mean = 71.53 [seventh grade], SE = 0.66, n = 30, P < .001). For the eighth-grade case,

English and Spanish outputs did not differ significantly in completeness or completeness ratio.; however, the Spanish eighth-grade cases were significantly more readable than those in English (English: mean = 64.22 [eighth to ninth grade], SE = 0.95, n = 30; Spanish: mean = 69.0 [eighth to ninth grade], SE = 0.82, n = 30, P < .001) (Table 4).

Table 4.

Completeness and readability in English versus Spanish for fifth-grade and eighth-grade cases.

Output attribute	English (n = 30)	Spanish (n = 30)	Difference
Completeness (fifth grade), mean ± SEM	60 ± 4%	51 ± 4%	t(58) = 1.61, P = 0.11
Completeness ratio (fifth grade), mean ± SEM	78 ± 6%	65 ± 6%	t(58) = 1.66, P = 0.10
Completeness (eighth grade), mean ± SEM	63 ± 3%	55 ± 3%	t(58) = 1.71, P = 0.09
Completeness ratio (eighth grade), mean ± SEM	82 ± 5%	71 ± 4%	t(58) = 1.72, P = 0.09
Readability (fifth grade), mean ± SEM	77.24 ± 0.98	71.53 ± 0.66	t(58) = 4.82, P < .001
Readability (eighth grade), mean ± SEM	64.22 ± 0.95	69.0 ± 0.82	t(58) = −3.82, P < .001

Model omissions

Elements that were missing across all discharge instructions were as follows:

• Diarrhea/vomiting prompt: return criterion of bloody or green vomit (no output mentioned green vomit specifically).

• Fever prompt: no output explicitly mentioned that antipyretics can reduce fever, but do not prevent it from returning.

Discussion

To our knowledge, this study is the first to assess generative large language models’ production of discharge instructions in English and Spanish. While GPT-4-generated readable discharge instructions with minimal prompting, outputs included only 57% of literature-derived completeness criteria across multiple discharge scenarios. Outputs were slightly more complete in English than in Spanish (62% versus 53%)., suggesting that the model is less competent at the discharge instruction generation task in Spanish than in English. In addition to GPT-4’s overall low completeness in the generation of discharge instructions, the model’s inferior performance in Spanish should also be an improvement target for developers and clinicians. As generative language models improve and individuals and organizations start turning to these tools for multi-lingual discharge instruction generation, the importance of these models’ multilingual performance will only continue to grow. One method to improve large language models’ performance in Spanish could be to increase the size of their Spanish-language medical training corpora: because GPT-4 is a transformer-style model which predicts the next token (eg, a word) in a series of text based on previous training data, differences in the size or quality of English- versus Spanish-language training data can introduce differences in output quality across languages.^29–32

An important caveat to consider when interpreting the current study’s results is the heterogeneity in clinician opinion on the most important items to include in discharge instructions. Unfortunately, there exist no specific guidelines on the return criteria to include in pediatric emergency room discharge instructions, as even the American Academy of Pediatrics Joint Policy Statement regarding guidelines for the care of children in the pediatric ED does not provide guidance of specific return criteria.²⁴ Indeed, the consensus on information to include in discharge instructions in the Delphi study used to determine return criteria ranged from 51% to 100%.²³ While improvements in pediatric ED guidelines may help resolve some of this ambiguity in the future, completeness figures must until then be interpreted with the knowledge that even an output containing all 5 return top criteria may be graded as under 100% complete by some clinicians.

Completeness scores found in this study (ranging from 52% to 62%) are in line with those in other recent studies. For example, in a study of the quality of Spanish-language GPT-4-generated answers to questions regarding chronic diseases, GPT-4-generated instructions were rated by clinicians as “comprehensive” ≤50% of the time, with the remainder being “Correct but inadequate” or containing information that was a mix of correct or incorrect.³³ In another study examining English-language GPT-4-generated answers to urology questions, physician-rated comprehensiveness was generally high, but 66% of the raters did not endorse high comprehensiveness for the emergency cases specifically.⁶ Therefore, GPT-4 performance for the discharge-instruction task seems to be in line with its performance on other clinical tasks.

In terms of readability, GPT-4 was successful in generating outputs at the eighth- but not fifth-grade level; specifically, the fifth-grade cases had a readability at the seventh-grade level, and the eighth-grade cases had a readability at the eighth- to ninth-grade levels. While seventh grade is at a higher level than that recommended by the American Medical Association,²⁷ a document written at a seventh-grade reading level is likely still accessible to the average emergency department patient,²² and is more accessible than most current discharge instructions. Older data suggest that ED discharge instructions have been written at approximately an 11th-grade reading level historically,³⁴ with more recent data from internal medicine services revealing an average discharge instruction reading level at the 10th- to 12th-grade reading level, with only 11.3% of discharge instructions being written below the seventh-grade reading level.³⁵

Furthermore, these results demonstrate the partial success of prompt engineering (ie, the practice of designing prompts for large language models to guide the model toward generating the desired output), as specification of reading level did lead GPT-4 to successfully modulate this parameter to some degree. Prior studies have also shown GPT-4’s ability to reduce medical consent forms from a college- to an eighth-grade reading level.³⁶ Interestingly, a separate large analysis of GPT-4-generated prompts indicated that the floor for GPT-4 in terms of reading level was sixth grade, which may explain the model’s inability to produce fifth-grade-level outputs.³⁷ This large study was also published in November 2023, 5 months after data generation for the current study, so GPT-4’s reading-level modulatory capabilities may have also improved over this time.

Of note, busy clinicians may not always include reading-level specifications; however, given that GPT-4 is able to intake custom instructions to be applied to all outputs, hospitals could set up GPT-4 (or other large language models) with a specification toward reading level to remove the need of clinicians to input this every time. Future work examining baseline reading level of GPT-4 outputs without guidance would also be illuminating.

Because previous research has shown success in using user input to direct the performance of transformer-based generative models,³⁸ the authors also briefly explored prompt engineering as a method to ameliorate these issues. Specifically, GPT-4 was given the same prompts in English and Spanish for the head injury with concussion case but instructed (in English) to include the specific completeness criteria examined in the present study. When this additional step was performed, both the English and Spanish outputs successfully incorporated these completeness criteria. Taken together, these findings suggest 2 parallel yet complementary avenues for further research: characterization of the differences in GPT-4 outputs based on language and reading level and further prompt reengineering to improve generated output. Preliminary research suggests GPT-4 might be used improve its own prompts,³⁹ which presents another potential avenue of study. Large language models may also be used to preemptively generate discharge instructions, allowing for pilot testing and storage of these instructions until they are needed.

Strengths of the current study include a standardized evaluation of multiple discharge instructions, allowing for a systematic and rigorous evaluation of GPT-4 produced outputs and providing future researchers with a template for similar analyses. This study also provides future researchers with a baseline of GPT-4’s performance on the generation of simple discharge instructions, which will be useful when evaluating the model’s performance on more complex tasks. Finally, this study highlights the importance for researchers and clinicians to approach generative AI technologies with thoughtfulness and care, as these technologies may inadvertently worsen disparities in researchers’ attempts to reduce them.

One limitation of this study is our approach to minimize prompt engineering. While this methodologic choice was intended to mimic a busy clinician quickly querying GPT-4 for discharge instructions, the authors acknowledge the large effects that prompt design can have on GPT-4 outputs⁴⁰ and welcome further research into applications of prompt engineering in healthcare document generation. A second limitation relates to the metrics used to analyze reading level that, while validated and widely used for this purpose, rely on such factors as number of syllables and number of words. These tools were chosen because of their prior use in medical contexts and because the Fernandez-Huerta score is a derivative of the Flesch-Kincaid score,²⁸ which allows for better comparability between the measures; however, they may miss more nuanced complexity in discharge instructions. Other health-literacy-related aspects of GPT-4-generated discharge instructions worth examining could include the use of jargon, the structural organization of discharge instructions, and patient-reported measures of comprehensibility and actionability. Finally, the novelty of the completeness ratio measure is a limitation; while the completeness ratio may prove a useful tool to contextualize completeness scores in the context of heterogenous clinical opinions, further validation of the score will be necessary to allow for full confidence in its use and interpretation. Especially given the ambiguous nature of some of the criteria used in the present study, further validation of essential return criteria will also be important in evaluating future attempts to generate complete discharge instructions.

These limitations provide numerous avenues for future work. Future studies should investigate how the quality of discharge instructions can be further improved using refined prompt tuning, both at the level of the individual user as well as automated prompt-editing innovations. More specifically, user-level interventions such as templates for produced text may improve GPT’s ability to produce outputs better in line with clinicians’ needs,⁴¹ and clinician input can be used to further evaluate the quality of discharge instructions. It will also be important to use other tools (eg, machine learning approaches that can take into account more complex structural features of texts)⁴² to compare reading level in terms of vocabulary and other text features. Finally, the relationship between objectively defined output completeness and physician-determined appropriateness of outputs can be further investigated to better delineate performance standards during the development of GPT-4 tools prior to wide-scale testing.

Future studies can also examine prompt engineering to determine the best way to phrase requests for discharge instructions in both languages. Studies of generalizability across other complex diseases or in new clinical settings are also important steps to better understand these tools in the clinical domain. At present, these tools are not yet sufficiently evaluated to be safe for clinical use without human-in-the-loop clinical editing. Prompts including personalized instructions might improve the likelihood that generated text include points critical for particular patients. Additionally, the generated outputs (in both languages) might serve as useful templates that can be minimally edited to provide useful discharge instructions for Spanish-speaking patients.

Conclusion

Busy clinicians in fast-paced clinical settings might reach for GPT-4 and similar tools to quickly produce clinical documentation including discharge instructions. These tools’ ability to produce outputs in numerous languages might serve to improve clinical workflows and reduce disparities at discharge; however, without prompt engineering, GPT-4 does not generate complete discharge instructions as of June 2023, a shortcoming which is more apparent in Spanish than in English. While the current study provides a baseline of GPT-4’s performance in basic cases of this task, careful and rigorous further evaluation of such tools both in simulation and in controlled clinical studies are still needed before they can be safely applied in the clinical space for more complex discharges.

Author contributions

Alex Gimeno conceived of and designed the study, generated, and analyzed the data, and helped write the manuscript. Kevin Krause aided with study design, document scoring, and writing the manuscript. Starina D’Souza assisted with document scoring and interpretation of results. Colin G. Walsh aided with study design and writing the manuscript.

Supplementary material

Supplementary material is available at JAMIA Open online.

Funding

C.G.W. is supported by funding from Wellcome Leap MCPsych; NIMH R01 MH121455; NIMH R01 MH120122; NIMH R01 MH118233; NIH 5RM1HG009034; and BRIDGE2AI (A.P.). K.K. is supported by funding from NLM Training Grant T15 LM007450.

Conflicts of interest

None of the authors have competing interests to declare.

Data availability

The data underlying this article are available in the following GitHub repository: https://github.com/AlexGimeno3/GPT_Corpus/.