Use and Utility of Patient After‐Visit Instructions at a University Rheumatology Outpatient Clinic: Status and Randomized Prospective Pilot Intervention Study

Abstract

Objective

The aim of this study was to evaluate the use of after‐visit instructions (AVIs) in an academic rheumatology clinic and assess the impact of standardized AVIs (sAVIs) and teach‐back (TB) on comprehension of health information.

Methods

A retrospective review of adult patients seen between October 1 and 8, 2021, at the rheumatology clinic collected data on patient demographics, clinical features, and the presence, content, and readability of AVIs. During a subsequent prospective proof‐of‐concept study, routinely scheduled patients seen at the rheumatology clinic were randomized into three groups: control (received standard of care), received sAVIs only, and received sAVIs plus TB. Patients completed a health literacy questionnaire, satisfaction survey, and a one‐ to two‐week postvisit telephone survey to assess AVI comprehension.

Results

Out of 316 retrospective patient visits, 82 (25.9%) received AVIs. Among 210 of 316 patients (66.5%) with management changes, 76 (36.1%) received AVI, with 74.2% of the instructions considered concordant with the provider's note. Use of AVIs was higher with management changes, new patient visits, and medical trainee/teaching clinics. AVIs were written at a median 6.8 grade level. A total of 75 patients completed the prospective study: 31 (41.3%) were in the control group, 19 (25.3%) were in the group that received sAVIs only, and 25 (33.3%) were in the group that received AVIs with TB. There were no differences in overall postvisit survey comprehension/retention scores among the three patient groups evaluated.

Conclusion

Although a lack of AVI use was identified, implementation of sAVIs did not appear to impact patient retention or comprehension of discharge health information.

INTRODUCTION

Patient education is an essential aspect of administering high‐quality care. How well patients understand their diagnoses and implement their plan of care is based on a multitude of factors that can drastically impact disease outcomes. ¹ , ² One of the key social determinants that affects a patient's ability to comprehend and act on their medical conditions is health literacy (HL). The Centers for Disease Control and Prevention (CDC) defines two different types of HL in their updated Healthy People 2030 initiative. Personal HL is “the degree to which individuals have the ability to find, understand, and use information and services to inform health‐related decisions and actions for themselves and others.” Organizational HL is “the degree to which organizations equitably enable individuals to find, understand, and use information and services to inform health‐related decisions and action for themselves and others.” These new definitions of HL now acknowledge that organizations have a responsibility, from a public health perspective, to address HL and optimize their approach to varying levels of personal HL. ³

SIGNIFICANCE & INNOVATIONS.

• After‐visit instructions are an underused resource in our academic clinic with around one‐fourth of patients receiving them.

• Interventions such as standardized after‐visit instructions and teach‐back did not appear to improve patient comprehension and retention of discharge information.

In 2003, the National Assessment of Adult Literacy performed a large‐scale survey of both literacy and HL. It showed that 36% of adults had basic or below‐basic HL skills. The patients that had the lowest HL tended to be older than 65 years old, Hispanic or Black, and male; have less than a high school education; and to be at or below the poverty line. ⁴ Low HL has been linked to poor health outcomes, including in patients with rheumatic diseases such as rheumatoid arthritis (RA) and systemic lupus erythematosus. ⁵ , ⁶ Various studies have been performed assessing literacy rates in rheumatology clinics using the tool the Rapid Estimate of Adult Literacy in Medicine (REALM). These studies have shown that a range of 10% to 19% of patients within diverse socioeconomic rheumatology outpatient settings have literacy less than a ninth grade reading level. ⁷ , ⁸ , ⁹ These studies were limited, however, by their predominantly White population. The American Medical Association (AMA) and the National Institutes of Health (NIH) have recommended that medical information provided to patients be written at a sixth grade level. ¹⁰ However, a 2019 study looking at patient educational materials (PEMs) for common rheumatic drugs from prominent rheumatology organizations in Canada, the United Kingdom, and Australia consistently showed readability scores higher than the ninth grade level. ¹¹ Another recent study evaluating the reading and comprehension of PEMs available on the American College of Rheumatology website showed that the majority were written at the 11th grade level. ¹² This indicates that PEMs provided by prominent rheumatology organizations are not in line with the HL guidelines recommended by the CDC, NIH, or AMA. However, the reading levels and comprehension of discharge instructions provided to patients in rheumatology clinics is not known.

Various strategies have been implemented to address the need for improving patients’ understanding of their medical conditions and management. In 2009, the United States federal government started incentivizing health care organizations to provide patients with after‐visit summaries (AVSs). The purpose of the AVS is to enhance information retention, support patient engagement, and improve quality of records. ¹³ Survey studies have shown that two‐thirds of patients referred to their AVS after their visit. ¹⁴ After‐visit instructions (AVIs) are more personalized instructions written by the patient's provider and are separate from the autogenerated AVSs that patients typically receive. In the emergency room setting, studies have shown that simplified and standardized discharge instructions can help increase patients’ overall understanding of their emergency room visit. ¹⁵ The “teach‐back” (TB) method, in which patients repeat instructions from the provider in their own words, is a strategy used to bridge gaps in HL. In the emergency room setting, TB has been shown to reduce revisits and result in greater retention of instructions. ¹⁶ TB communication in a safety‐net hospital setting rheumatology clinic has also been shown to improve overall medication compliance. ¹⁷ Although TB has been assessed in other settings, the previous study was the only assessment of TB in a rheumatology setting, and TB has never been compared to standardized AVIs (sAVIs). This is an important knowledge gap to address in order to identify the most time‐efficient ways to deliver and improve patient postvisit education.

With the goal of optimizing patients’ health outcomes through efforts focused on organizational HL, we undertook a two‐part study. First, we conducted a needs assessment evaluating the use and appropriateness of AVIs provided to patients in an academic rheumatology outpatient setting. Second, in a proof‐of‐concept setting, we looked at the effectiveness of two AVI interventions aimed at improving comprehension and retention of management plans by patients in a rheumatology outpatient setting.

PATIENTS AND METHODS

Phase 1: needs assessment

A retrospective study was performed as a preliminary needs assessment to evaluate the use of AVIs in the Rush University Medical Center Rheumatology Clinic (RUMCRC). We hypothesized that the majority of patients that had changes in management would not receive AVIs. This study spanned from October 1, 2021 to October 8, 2022, and included patients older than 18 years of age. Telemedicine visits were excluded given the challenges of implementing instructions during these visits.

The following data were collected: basic demographics, rheumatologic diagnoses, medications, comorbidities, presence of AVIs, readability of AVIs using Flesch–Kincaid scores, components covered within AVIs (diagnosis, medications, management plan), care provider information, author of AVIs, the patients’ documented preferences for learning, and patient‐reported outcome questionnaire data assessed by the multidimensional health assessment questionnaire (MDHAQ).

The primary outcome was the frequency of provision of AVIs at discharge among all patients seen at the RUMCRC. The following correlates of AVI provision were evaluated: visits with or without a change in management, inflammatory versus noninflammatory primary diagnosis, new (first two) versus established (more than two) patient visits, Charlson Comorbidity Index, tiers of rheumatologic medications, patients’ preferred learning modality, primary language, and if patients were seen in a teaching clinic or solo attending clinics. Rheumatologic medications were stratified into four tiers based on side effects, drug monitoring, and infection risk, with the higher tiers indicating greater risk (Supplementary Table 1).

Secondary outcomes included the readability of AVIs and concordance between the AVIs and the providers’ documentation of the management plans. Readability was evaluated using the Flesch–Kincaid reading score and grade level, a validated readability test that assesses syllables, words, and sentences to determine the ease of readability of a given passage. The website used to perform these tests was WebFX. ¹⁸

Correspondence between the changes in management documented in the provider's note and those documented in the AVIs were assessed. Three components of management changes were evaluated: medications changes, relevant disease education, and additional diagnostic or subspecialty workup. Changes in management were defined as the following: education on new (or change in) diagnosis, new medications or any change to an existing medication, referral to a subspecialty or follow‐up with a subspecialty, new lifestyle education modification, or diagnostic testing that required future scheduling. Correspondence was calculated as an aggregated percentage based on the above three components. SPSS software was used for data analysis. Statistical analyses performed included descriptive statistics, chi‐square analysis, and Student's t‐tests to compare categorical and continuous data. A P value of ≤0.05 was considered significant on Student's two‐tailed t‐tests.

Phase 2: randomized prospective comparison of sAVIs and TB

This was a prospective, randomized, proof‐of‐concept, three‐arm study aimed at optimizing AVI delivery in an academic outpatient rheumatology setting by implementing strategies such as patient sAVIs and TB. The prospective aspect of this study was reviewed and approved by the Institutional Review Board at the Rush University Medical Center. The inclusion criteria were patients who were older than 18 years of age and patients seen in the RUMCRC. The exclusion criteria were telemedicine visits and English not being the primary language. This was due to the inability of all providers to provide custom AVIs due to language limitations.

The three arms were as follows: the control group, the group that received sAVIs only, and the group that received sAVIs and TB. The control group included patients receiving the current standard of care at the RUMCRC. The group that received sAVIs only included patients receiving typed sAVIs, which included a text template (Supplementary Figure 1) that providers used to write customized AVIs for the patient. The group that received sAVIs and TB included standardized written AVIs with the patient repeating to the provider which changes and future management had been agreed upon during the visit. All participating providers were individually educated using a standardized framework of instructions on how to implement each strategy before the study.

The study aimed to assess the effectiveness of these strategies by measuring patient retention and comprehension of discharge information as the primary outcome. The secondary outcome was patient satisfaction. Outcomes were measured with postvisit surveys that focused on assessing satisfaction (Supplementary Figure 2) and testing retention and comprehension (Supplementary Figure 3). The study sequence for the prospective portion is outlined in Figure 1.

Figure 1.

Intervention Study Sequence of Events

Patients participating in this study were recruited from the RUMCRC, where they were seen by solo‐care providers (nontrainee clinics) or in the fellow continuity clinic. Participating providers were chosen to reflect diversity in gender, age, and experience. The following baseline data points were collected for patients in all cohorts: basic demographics, new patients versus follow‐up visit, MDHAQ scores, Arthritis REALM (A‐REALM) score, rheumatologic diagnosis, medications being received, Charlson Comorbidity Index, barriers to learning, and patient preferences for learning.

During the postvisit telephone survey, patients answered questions to assess their comprehension and retention of information provided during their clinic visit regarding their disease and management plans. Their responses were then compared to the health care providers’ notes, and a concordance score was calculated, yielding a percentage score. In addition to an overall concordance percentage score, we also performed a subgroup analysis in which we stratified groups by extent/degree of concordance as follows: complete (>75%), most (50%–75%), partial (25%–50%), and minimal (<25%).

After completion of the study, the reading and comprehensibility of the AVIs was assessed using Flesch–Kincaid readability scores and grade levels to see how they corresponded to patients baseline literacy. Provider surveys were used to determine the feasibility of implementing these discharge strategies in our rheumatology clinic. SPSS software was used for data management and analysis. Descriptive statistics were obtained. Retention and satisfaction scores in the three arms were compared using analysis of variance. A P value of ≤0.05 was considered significant on Student's two‐tailed t‐tests.

RESULTS

Phase 1: needs assessment

A total of 316 patients met the inclusion criteria for the first phase of the study. Most patients were female (78.2%), had English as their primary language (86.7%), and had a mean (±SD) age of 56.8 (±14.1) years (Table 1). Less than one‐third (31.6%) were new patient visits. Over 60% of patients had inflammatory or autoimmune primary rheumatologic diagnoses. Overall, 82 of 316 patients (25.9%) received an AVI during routine visits. Of the 210 (66.5%) patients that met the definition of change in management, 76 (36.1%) received an AVI.

Table 1.

Baseline demographics and characteristics of patients in the retrospective study^*

Characteristic	Total (n = 316)
Mean (±SD) age, y	56.8 (14.9)
Female, n (%)	247 (78.2)
English as primary language, n (%)	274 (86.7)
Ethnicity, n (%)
Hispanic	84 (26.8)
Not Hispanic	229 (73.1)
Race, n (%)
White	135 (43.2)
Black	101 (32.4)
Asian	8 (2.5)
Other	68 (21.7)
Established PCP, n (%)	289 (91.4)
New patient visit, n (%)	100 (31.6)
More than one clinic no show in the previous two years, n (%)	180 (57.0)
Primary rheumatologic diagnosis, n (%)
Inflammatory/autoimmune	192 (60.7)
Noninflammatory conditions	124 (39.2)
Mean Charlson Comorbidity (±SD) score a	2.6 (1.9)
Charlson Comorbidity score ≥2	211 (66.8)
Patients with ≥11 total medications, n (%)	161 (50.9)
Patients with tier 3 or 4 medications b	166 (52.5)
Clinic setting, n (%)
Medical trainee/teaching clinic	101 (32.0)
Solo attending	215 (68.0)
Patient preferred learning modality, n (%) c
No preference	169 (53.5)
Reading	75 (23.7)
Visual or reading barrier to learning, n (%)	7 (2.2)
AVIs provided, n (%)	82 (25.9)
Change in management during visit, n (%)	210 (66.5)
Change in management and received AVIs, n (%)	76/210 (36.1)

^*AVI, after‐visit instruction; PCP, primary care provider.

^a The Charlson Comorbidity Index provides a 10‐year mortality for patients with a range of comorbid conditions.

^b Tiers of rheumatologic medications were based on side effects, drug monitoring, and infection risk, with higher tiers conferring higher risk.

^c Learning modalities were recorded in the Rush Rheumatology Clinic electronic medical record.

Most patients received free text instructions (80%), followed by educational resources (15.9%) and both (3.7%). Of the patients that received AVIs, 74.2% of the instructions were concordant with the changes outlined in the provider's note. The most concordant category was “medication changes” (94.2%), followed by “diagnostic workup/follow‐up” (77.6%), with “disease education” being the least concordant (43.9%). The median (interquartile range) Flesch–Kincaid reading ease score and grade level of AVIs provided was 70.2 (±20.9) and 6.8 (±4.6), respectively (Table 2).

Table 2.

Characteristics of AVIs in the retrospective study^*

Characteristic	Total (n = 82)
Type of AVI, n (%)
Only free text	66 (80.5)
Only educational resource	13 (15.9)
Both free text and resource	3 (3.7)
AVI appropriateness, n (%) a
Relevant disease education	29/66 (43.9)
Medication instructions	65/69 (94.2)
Diagnostic workup and follow‐up	52/67 (77.6)
Concordance rate between AVIs and provider's note, %	74.20
Reading and comprehension of AVIs, mean (±SD), median (IQR) b
Flesch–Kincaid reading ease score	67.6 (±18.21), 70.2 (21.9)
Flesch–Kincaid grade level	7.1 (±3.35), 6.8 (4.6)
AVI written by trainee, n (%)	30/82 (36.5)

^*AVI, after‐visit instruction; IQR, interquartile range.

^a These three categories were identified by the investigators from the provider's note and then compared to the AVIs.

^b Flesch–Kincaid tests are readability assessments that are widely used to evaluate the reading ease and grade level of a given text. The higher the reading ease score, the easier the text is to understand, and a score of ≥70 is recommended.

Higher frequency of AVI use was noted in patients that had a change in management (P < 0.001), new patient visits (P = 0.002), and if patients were seen in a medical trainee/teaching clinic (P = 0.007). There were no differences in AVI frequency based on the patients’ primary language, inflammatory versus noninflammatory rheumatologic condition, presence of higher risk in third/fourth tier medications, number of medications, Charlson comorbidity score, or learning preferences/barriers (Supplementary Table 2).

Phase 2: randomized prospective comparison of sAVIs and TB

Out of 510 patients who were contacted to participate in this study, 120 patients were randomized into the prospective arm of the study during the study period December 19, 2022, to April 21, 2023, with 75 patients completing the study in its entirety (Supplementary Figure 4). Baseline characteristics of these 75 patients are presented in Table 3. Most patients were female (75.7%) and Black (45.3%). The mean (±SD) age of patients was 56.7 (±14.5) years. New patients comprised 40% (30 of 75) of the entire cohort. Mean (±SD) years of education in this cohort was 15.08 (±2.97) years, and mean (±SD) A‐REALM score was 61.83 (±6.85), which equates to a high school HL level. Out of the 120 randomized patients, 28 of 120 (23.3%) had A‐REALM scores under ninth grade level. Out of the 75 patients that completed the study, 13 of 75 (17.3%) had A‐REALM scores under ninth grade level. Nearly half of the patients had an inflammatory or autoimmune primary rheumatologic diagnosis.

Table 3.

Baseline demographics and characteristics in the intervention study^*

Characteristic	Total (n = 75)
Mean (±SD) age, y	56.7 (±14.5)
Female, n (%)	56 (75.7)
Ethnicity, n (%)
Hispanic	13 (17.3)
Not Hispanic	62 (82.7)
Race, n (%)
White	27 (36)
Black	34 (45.3)
Asian	2 (2.7)
Other	11 (14.7)
New patient, n (%)	30 (40)
Mean education (±SD), y	15.08 (±2.97)
Mean A‐REALM (±SD)	61.83 (±6.85)
Primary rheumatologic diagnosis, n (%)
Inflammatory/autoimmune	35 (47.3)
Noninflammatory conditions	36 (48.6)
Not applicable	4 (5.3)
Study arm, n (%)
Control	31 (41.3)
AVIs only	19 (25.3)
AVIs and TB	25 (33.3)
Change in management during visit	64 (85.3)
AVIs provided	57 (77)
AVIs provided in control group	14/31 (45.2)
AVIs provided in control group with change in management	13/25 (52)

^*A‐REALM, Arthritis Rapid Estimate of Adult Literacy in Medicine; AVI, after‐visit instruction; TB, teach‐back.

Of patients who completed the study in its entirety, 31 (41.3%) were in the control group, 19 (25.3%) were in the group that received sAVIs only, and 25 (33.3%) were in the group that received sAVIs with TB. All patients in the intervention groups received AVIs, whereas 45.2% (14 of 31) patients in the control group received AVIs. There were no differences in baseline characteristics among the three groups.

For primary outcome analysis, we evaluated the postvisit retention survey responses and compared scores across three study groups in the three evaluated categories. There were no statistically significant differences in the total number of patients achieving high comprehension/retention scores in three groups receiving AVIs and no difference in concordance across any of the categories evaluated: reason for visit (P = 0.399), medication changes/side effects (P = 0.270), and referrals (P = 0.631; Table 4). On subset analysis, when comparing patients that had complete concordance scores (defined as >75%) in their respective categories, patients in the group that received TB and sAVIs had significantly higher concordance in the medication changes/side effects (P = 0.032) category when compared to the control group or the group that received sAVIs only. Otherwise, there was no difference across reasons for the visit or referrals categories when assessing concordance >75% (Supplementary Table 3).

Table 4.

Patient concordance on the postvisit retention survey in the intervention study^*

Survey	Control (n = 31)	AVIs only (n = 19)	AVIs and TB (n = 25)	Chi‐square P value
Reason for visit
Complete	18	16	18	0.399
Most	9	3	4	0.399
Partial	3	0	3	0.399
Minimal	1	0	0	0.399
Medication changes/side effects
Complete	11	7	17	0.270
Most	4	7	4	0.270
Partial	1	3	1	0.270
Minimal	3	2	0	0.270
Referrals and diagnostics
Complete	12	10	13	0.631
Most	2	1	1	0.631
Partial	1	3	1	0.631
Minimal	2	0	1	0.631

^*AVI, after‐visit instruction; TB, teach‐back.

Of the 75 patients who completed the study, 54 completed the satisfaction survey: 22 of 31(71.0%) in the control group, 14 of 19 (73.7%) in the group that received sAVIs only, and 18 of 25 (72%) in the group that received sAVIs and TB. In the satisfaction survey, a higher number of patients in the control group answered “strongly agree” to questions regarding whether they felt their doctor was easy to understand (P = 0.004) or if their doctor showed them respect (P = 0.025). Responses to the other questions in the satisfaction survey showed no differences among the three groups. (Table 5). When both the intervention groups were combined and then compared to the control group, satisfaction survey results showed significantly higher scores to specific satisfaction questions regarding how their doctor explained management to them (P = 0.007) and if they felt respected by their doctor (P = 0.023) in favor of the combined intervention groups.

Table 5.

Satisfaction survey results of patients that answered strongly agree in the intervention study^*

Survey response	Control (n = 22), n (%)	AVIs only (n = 14), n (%)	AVIs and TB (n = 18), n (%)	Chi‐squared P value
Easy to understand	21 (95.4)	7 (50)	16 (88.8)	0.004
Listened carefully	18 (81.8)	7 (50)	16 (88.8)	0.139
Showed respect	21 (95.4)	10 (71.4)	15 (83.3)	0.025
Spent enough time	18 (81.8)	10 (71.4)	16 (88.8)	0.557
Confident in provider's ability	19 (86.3)	7 (50)	14 (77.8)	0.168

Note: Bolded P values are statistically significant (P < 0.05).

^*AVI, after‐visit instruction; TB, teach‐back.

The mean (±SD) Flesch–Kincaid readability scores for discharge instructions were 66.14 (±11.88) in the control group, 47.59 (±11.96) in the group that received sAVIs only, and 48.45 (±12.34) in the group that received sAVIs and TB (P <0.01; higher scores equate to greater ease of reading). The mean (±SD) Flesch–Kincaid readability grade levels were 7.6 (±3.25) in the control group, 9.2 (±1.90) in the group that received sAVIs only, and 8.8 (±1.81) in the group that received sAVIs and TB (P = 0.118; Supplementary Table 4). Of the patients who were provided AVIs, as evidenced by documentation in the electronic medical record (EMR), only 71.9% (41 of 57) stated that they received the AVIs either physically after their visit or through the patient portal. Of the 41 patients who endorsed receiving AVIs, 70.7% (29 of 41) of patients stated that they read the AVIs. Approximately 20% (4 of 19) of patients in the group that received sAVIs only and 40% (10 of 25) in the group that received sAVIs and TB reported not having received AVIs. (Supplementary Figure 5)

A total of nine providers participated in this study. A total of 56% (five) of providers agreed that sAVIs improved patient comprehension and were a valuable use of clinical time. Regarding TB, most providers felt neutral about its impact on patient comprehension (n = 5, 56%) and its effectiveness in using clinical time (n = 6, 67%). Among the providers, 56% (n = 5) reported that typing the AVIs took one to two minutes, and 56% (n = 5) of providers expressed that implementing TB required two to four minutes. With regard to use in their future clinical practice, only three of nine (33.3%) providers stated they were “likely” to use sAVIs, and two of nine (22.2%) providers stated they were “likely” to use TB.

DISCUSSION

Low HL is a significant social determinant that hampers patients’ understanding and adherence to the management of their medical conditions. Organizational HL refers to the extent to which organizations effectively meet the demand for accessible, comprehensible, and valuable information and services that empower patients to make informed decisions. The delivery of comprehensive and easy‐to‐read AVIs is potentially one way to improve organizational HL. In the initial phase of our study, we evaluated the use and appropriateness of AVIs given to patients in an academic rheumatology outpatient setting. We identified a gap in the delivery of AVIs with only one‐fourth of patients in our clinic receiving any AVIs and with only one‐third of patients who had a change in management receiving AVIs. When instructions were provided, however, they were mostly concordant with what was discussed during the visit and were written at a reasonable seventh grade level.

In efforts to improve HL and optimize AVI delivery, we conducted a prospective, proof‐of‐concept comparative study aimed at evaluating whether strategies such as sAVIs and TB could improve patients’ understanding of their disease and management plans when compared against standard of care. Neither the presence of AVIs nor the implementation of an optimized AVI process had a significant impact on our patients’ retention or comprehension of the discharge health information provided to them. A similar study was conducted in the primary care setting where patients were randomized to receive various levels of AVS content: minimum, intermediate, maximum, or standard AVS. There were no differences in patient recall or patient satisfaction levels among all groups and HL levels. ¹⁹

In contrast, studies in the emergency room setting have shown improved retention with standardized discharge instructions. ¹⁵ The acuity of disease and the acute setting may select for a different patient population than our clinic and motivate patients to reference education materials more frequently. Prior studies have also shown that TB improves medication adherence in patients seen at the rheumatology clinic. ¹⁷ Although our study did not necessarily evaluate medication adherence, it is worth noting that patients in the group that received sAVIs and TB had significantly higher levels of complete (>75%) concordance in the medication changes category when compared to the other groups.

Unexpectedly, the control group, in which more than half the patients did not receive AVIs, reported higher satisfaction scores in two survey responses compared to the active intervention groups. These responses assessed whether the patients felt their provider was easy to understand and if they felt respected by their provider. A possible explanation for this is that providers in the control group engaged in more direct verbal communication with patients, spending additional time addressing their concerns, whereas providers in the intervention groups spent less “face‐to‐face” time with patients when they were using the EMR to formulate discharge instructions. However, previous systematic reviews have shown that patient satisfaction is not impacted by the use of EMRs in direct patient care, but they did not specifically address AVIs. ²⁰

Given the small numbers of patients in each of the groups that received sAVIs, we combined the intervention groups together and then compared patient satisfaction responses to those seen in the control group. Satisfaction scores regarding how doctors explained concepts to them and whether patients felt respected were significantly higher in the intervention groups. This could possibly indicate the effect of small sample size in each group, impacting the results (type 2 error). A larger study may address this issue more clearly.

Though only 45.2% of patients in the control group received any AVIs, they had better readability scores than the intervention groups. When assessing our template, we identified certain words like “rheumatology” and “medication” that, due to the number of syllables, negatively impacted the Flesch–Kincaid readability score. Additionally, given that our template was comprehensive, it was much longer than the typically concise free text instructions provided in the control group. Although the readability scores favored the control group, there was no significant difference in the Flesch–Kincaid grade level, which assesses comprehension level.

The readability scores for AVIs in the two intervention groups were at ninth grade level, exceeding the sixth grade reading level recommended by the AMA and NIH. These findings are in keeping with another study that showed that popular patient education materials in rheumatology were around ninth grade or higher reading levels using Flesch–Kincaid scores. ¹¹ In our study, 23% of patients had a ninth grade or lower reading level based on the A‐REALM. This percentage of patients is higher than the 10.3% to 19% (ninth grade or lower) previously reported in other studies from patients seen in rheumatology clinics in rural, metropolitan, and academic settings. ⁷ , ⁸ , ⁹

Because names of medications and diseases can drastically increase the reading level of a text, a more accurate assessment of Flesch–Kincaid reading level may be calculated by excluding names of medications and diseases to avoid the influence of special medical terminology. We also suggest using the correct scientific diagnosis (eg, RA) at the beginning of the patient information and then using a simpler term like “this condition,” “this medicine,” to make AVIs more patient friendly.

When analyzing provider responses on feasibility and satisfaction, it was found that sAVIs were less time intensive to implement than TB. This was reflected by 56% (five of nine) of providers who believed that sAVIs were a good use of clinical time, whereas only 22% (two of nine) thought TB was a good use of clinical time. More providers stated they were unlikely to continue to provide TB than sAVIs in their future clinical practice. Actual implementation time was not assessed to prevent any negative impact on the patient and physician interaction.

There are several limitations of this study. Only 70% of patients who received formulated AVIs confirmed having access to them. Of note, providers did not directly deliver to or review the printed AVIs with patients. Instead, AVIs were delivered by the clinic coordinators at discharge. It came to our attention that patients were sometimes given the choice to have their instructions printed at checkout, and those who declined did not receive the AVIs. Some patients may have missed the AVIs “buried” in the discharge package, and others may have chosen not to have them printed, intending to refer to the AVIs on the My Chart portal but failed to check it. Given that 30% of patients were unaware of AVIs, this could have impacted the results. Patients, though randomized, were enrolled from a convenience sample of patients attending the rheumatology clinic. There were many patients who were contacted but declined to take part, and it is possible that patients who agreed to participate may have been different from those who declined. Of the 123 people who agreed to partake and were randomized, only 75 (60.9%) completed the study due to no shows, provider error, or incomplete telephone survey. We also excluded non‐English speaking patients, an important patient demographic with different HL needs. Also, most patients in our study were female and Black, reflective of the demographic makeup of our rheumatology clinic. This makes the results less generalizable.

The telephone survey tool we used to assess concordance has not previously been validated and involved a degree of subjective measurement, therefore potentially making it less reliable. However, we used the same study personnel to perform telephone surveys to try and mitigate variance in concordance. We also acknowledge that we did not evaluate other important methods of patient discharge information such as printed pamphlets or other web‐based AVI patient education materials that are provided to patients outside of the EMR. Ultimately, this was a proof‐of‐concept study with a limited sample size, which can make it challenging to make conclusions to a larger general population, but it has identified areas for future study and clarification.

The strengths of this study include its randomized nature, which helps to distribute any confounding variables equally, and this is evidenced by the evenly distributed baseline characteristics of our patient groups (Table 3). Our study is unique in that we used the A‐REALM tool and Flesch–Kincaid score to correlate reading level with the grade level of written instructions. This identified an important gap between the readability of provided AVIs and the recommendations set forth by national organizations. Our study also identified areas for improvement, including the lack of processes that effectively highlight AVIs for patients (using a different color paper for example) and instances in which patients did not receive AVIs. Recognizing these areas of improvement will enable the implementation of strategies to ensure that patients receive and are aware of instructions given to them by their providers. Additionally, we believe that the gaps uncovered in this study and suggested areas of improvement may be applicable to other academic rheumatology institutions.

In summary, personalized AVIs are an underused tool in our academic rheumatology outpatient clinic, with just over one‐fourth of patients receiving them. However, in our pilot study aimed at optimizing discharge patient education, the implementation of sAVIs and TB did not appear to improve patient retention or comprehension of discharge health information across all groups. However, in a subset analysis of patients with complete concordance (>75%), patients in the group that received AVIs and TB did have higher concordance in medication changes than controls or patients who received sAVIs only. Although readability of AVIs was worse in the intervention groups, the instructions in the intervention groups tended to be more comprehensive. Our providers felt that AVIs were of more clinical value and a more efficient use of time than TB.

Future implementation of artificial intelligence and the use of large language models could possibly revolutionize the approach to AVIs. By incorporating these advancements, AVIs could automatically generate a summary from the providers’ assessment and plan at an appropriate reading level. This would potentially save time for providers because they would proofread the summarized materials rather than developing the instructions from scratch. Further research addressing the limitations of this study could allow us to understand the utility of these patient education strategies to optimize the clinic visit and health outcomes.

AUTHOR CONTRIBUTIONS

All authors were involved in drafting the article or revising if critically for important intellectual content, and all authors approved the final version to be published. Dr Hassan had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study conception and design

Joseph, Hu, Min, Jolly, Hassan.

Acquisition of data

Dijo, Rong, Robert.

Analysis and interpretation of data

Dijo, Jolly, Hassan.

Supporting information

Disclosure form

Supplemental Table 1: Medication Tiers Based on Side Effect Profile and Medication Monitoring

Supplemental Table 2: Frequency (n) of After‐Visit Instructions in Relation to Baseline Characteristics In the Retrospective Study

Supplement Table 3: Patients with Complete (>75%) Concordance on the Post‐visit Retention Survey in the Intervention Study

Supplemental Table 4: Mean (SD) Flesch Kincaid Readability Scores and Grade Levels For the Intervention Study

Supplement Figure 1: After Visit Instructions Template

Supplemental Figure 2: Post Visit Satisfaction Survey

Supplemental Figure 3: Telephone Survey Questionnaire and protocol

Supplement Figure 4: Flow Chart of Patient Consent, Randomization, and Completion of Study

Supplement Figure 5: Sankey Diagram of Patients Who Stated That They Received and Read After Visit Instructions