Evaluation of a Longitudinal Medical School Health Literacy Curriculum

Abstract

Background:

Despite recommendations that medical schools incorporate health literacy (HL) into curricula and identification of consensus areas of HL competence, high-quality data are needed for curricular characteristics and structured evaluation that foster sustained HL competency adoption.

Objective:

This study aimed to develop and evaluate a comprehensive longitudinal medical school HL curriculum using qualitative and quantitative assessments. We sought to ground this in existing theory and provide evidence for generalizable use and further theory refinement.

Methods:

Across three medical student cohorts, HL was integrated into a 14-month pre-clinical professional development course. The longitudinal curriculum was informed by consensus-derived HL competencies and Bloom's Taxonomy. Student self-assessment and reflection data were linked across three timepoints and analyzed using mixed methods: an inductive approach identified key qualitative themes; exploratory factor analysis (EFA) identified prevalent factors within self-assessments; and analysis of variance identified differences across timepoints.

Key Results:

Three qualitative themes emerged from student reflections: emotions associated with a backward reading exercise; shifts in awareness of HL as a patient challenge; and plans to continue using HL practices. Among 336 students with quantitative data across all timepoints, EFA identified three factors: foundations, shame-free environment, conveying information. Over the curriculum, students demonstrated significant (p < .05) improvements in each factor.

Conclusions:

Our longitudinal HL curriculum, grounded in existing competencies and conceptual framework, elicited positive changes related to medical student HL competencies. Qualitative data demonstrating motivation and intention to continue applying HL practices were augmented by quantitative data showing increased adoption of self-reported behaviors over curricular timepoints. This study fulfills multiple features of a conceptual framework for HL curricula in health professions education, including sequenced, interactive sessions, multiple instruction modes, reflection, integration of knowledge and skill education, and varied assessment methods. Our findings can be used by investigators, institutions, and professional accreditation organizations to broadly enhance HL education.

Plain language summary

Plain Language Summary: A 14-month engaging medical school HL curriculum with varied topics, teaching approaches, and assignments is described. Multiple evaluation methods showed significant increases in HL awareness, self-reported HL skills use, and plans to continue this in future roles. Results may be used to improve HL education, training, and supportive infrastructure, organizational HL efforts, and health outcomes.

“Achieving health and well-being requires eliminating health disparities, achieving health equity, and attaining health literacy” (Healthy People 2030, n.d.). The Institute of Medicine found that health literacy (HL) is fundamental to quality care—safety, patient-centered care, and equitable treatment, and that health professionals have limited education, training, and practice opportunities to develop skills for improving HL. They recommended health professional schools incorporate HL into curricula and competence areas (Institute of Medicine, 2004). HL education and training described in undergraduate and postgraduate medical and other health professions settings have heterogeneous educational and evaluation methods and conceptual underpinnings, and research needs have been identified to refine what should be taught and how; evaluate effectiveness; and assess impact on health outcomes (Abrams et al., 2023; Agency for Healthcare Research and Quality, (2024); Centers for Disease Control and Prevention, (2024); Coleman, 2011; Coleman et al., 2017; Coleman et al., 2022; Coleman & Fromer, 2015; Maybury et al., 2024; Saunders et al., 2019).

In 2017, Coleman et al. prioritized an earlier consensus list of health professionals' HL practices and educational competencies and identified a core set of practices they should learn and use to improve clinical outcomes. These have been applied to HL curricula for health professions students and informed assessment tools for HL behaviors. They have not been reliably considered from a structured educational framework like Bloom's Taxonomy that considers educational objectives across two dimensions: knowledge and cognitive process (Anderson & Krathwohl, 2001). Organizing consensus HL competencies into an accepted educational framework can facilitate more generalizable and reproducible curriculum interventions.

In a 2019 systematic review of studies of HL education for health professions students, Saunders et al. noted absence of a “coordinated approach to measurable competencies and assessment.” In 2024, Maybury et al. reviewed how HL is taught and evaluated in seven health professional and adjacent disciplines, finding HL education varies in quality, quantity, timing, and modes of education and evaluation; they recommended HL teaching as a strategy to “close gaps in patient/client professional communication for future generations of graduates and the people they serve.”

Reports of medical school HL curricula largely describe one or more didactic presentations or workshops with associated activities, and varied assessment of pre-/post-knowledge, confidence, attitudes, and intention to use (Maybury et al., 2024; Saunders et al., 2019). Evaluations demonstrate short-term improvement, but evidence for retention and application of HL-related skills is limited (Coleman et al., 2016). In 2022, Coleman et al. described an 18-month medical school longitudinal curriculum resulting in students establishing a question-eliciting communication habit. Data are needed on medical school HL education characteristics and effectiveness leading to sustained HL skills adoption.

To address this need at our own institution, we introduced a longitudinal HL curricular thread into a medical school small group professional development course. This study draws from existing conceptual framework and consensus-driven competencies to develop and evaluate this curriculum. We assessed changes in student knowledge, attitudes, and self-reported practices over the 14-month curriculum using mixed qualitative and quantitative methods. Given the dearth of evidence connecting consensus-driven concepts in HL education to investigator-reported empirical findings, it is important to ensure assumptions underlying the conceptual framework correspond with learner-centered perceptions and connections. Mixed methods allow evaluation of semi-structured learner-centered outcomes alongside more rigidly structured quantitative outcomes based on existing knowledge. We aimed to develop and provide validity evidence for evaluation methods grounded in existing theory to promote generalizability of the curriculum and its assessment tools by other investigators to enhance HL education and training broadly.

Methods

Setting

This study took place at a large midwestern medical school with a diverse student body and annual enrollment of about 200 students (The Ohio State University College of Medicine, n.d.a.). Its curriculum includes a Longitudinal Group (LG) professional development course for all first- and second-year medical students meeting 3 hours weekly to discuss interpersonal communication, physical examination, behavioral/social sciences, and clinical reasoning (The Ohio State University College of Medicine, n.d.b.). Sessions are led by faculty facilitators with small group didactics (8–12 students), supervised interviewing and physical examination practice, individual and group activities, and student-led presentations.

This study, approved by the University's Institutional Research Board (#2022E0157), includes three student cohorts from academic years 2019 to 2020 through 2021 to 2022. Among 617 enrolled students, 517 (84%) consented to educational data collection through the Undergraduate Medical Education Data Repository; those data were available for analyses.

Curricular Objectives, Activities, and Evaluation

Beginning in 2019 to 2020, five topic-specific HL sessions were integrated across the 14-month LG timeline (Table 1). Curricular objectives, activities, and assessments were informed by consensus-derived HL competencies (Coleman et al., 2017); revised Bloom's Taxonomy (Anderson & Krathwohl, 2001); and an evaluation tool for a competency subset with validity evidence (Abrams et al., 2023). Students prepared with pre-work. Facilitators prepared through a HL faculty development session, teaching notes and resources, and micro-teaching scripts tailored to curricular topics (Morris et al., 2021). HL session lengths were 45 to 60 minutes. Students completed periodic online self-assessments (Table A) and reflections via Qualtrics software (Qualtrics software, August 2023). Data were de-identified and linked across timepoints using randomly generated research IDs.

Table 1.

Health Literacy in Longitudinal Group Curriculum Summary

Session Topic(s) and Content	Learning Objectives	Student Pre-Work (≤30 minutes)	Session Activities	Evaluation
Year 1
Overview: definition; prevalence of and contributors to low HL; associations between low HL and health-related outcomes; awareness, recognition, and empathy	Define HL	Watch video: “Health literacy and patient safety: Help patients understand” (American Medical Association Foundation, 2010)	Adapted backward reading exercise (students read aloud paragraph with words written backward to simulate experience of limited literacy […naelc eht epat sdaeh dna natspac revenehw uoy eciton…]) (Weiss, 2007)	Baseline quantitative self-assessmenta
	Describe and recognize risk factors for, and demographics and prevalence of low HL.			Baseline qualitative reflection
	Describe associations between low HL and health-related outcomes
	Recognize and describe what it is like to struggle with limited literacy and/or understanding; and challenges related to limited HL
Universal Precautions (using clear communication tools with everyone) approach; introduction to plain language	Understand complexities related to screening for limited HL in the clinical setting	Read: Health Literacy 2.0: Integrating patient health literacy screening with universal precautions (Hadden & Kripalani, 2019)	Practicing plain language principles activity: identify key messages and translate into plain language	-
	Describe the Universal Precautions approach to HL and its rationale
	Recognize and apply key components of using plain language in verbal communication	Model and Use Plain Language, from Building Health Literate Organizations: A Guidebook to Achieving Organizational Change (Abrams et al., 2014)
Year 2
Plain language and Teach Back	Explain why HL practices (plain language and Teach Back) are important to safe high-quality care	Complete Always Use Teach-back! Toolkit (AUTB) Teach-back Interactive Learning Module (Institute for Healthcare Advancement, 2024a)	Teach-Back role-play using AUTB 10 Elements of Competence for Using Teach-back Effectively and Teach-Back Observation Tool (10 Elements of Competence for Using Teach-back Effectively) (Institute for Healthcare Advancement, 2024a; Institute for Healthcare Advancement, 2024b)	Midpoint quantitative self-assessmenta
	Describe key elements of using Teach Back well
	Use Teach Back to assess and ensure patient/family understanding

Session Topic(s) and Content	Learning Objectives	Student Pre-Work (≤30 minutes)	Session Activities	Evaluation
Reader-friendly plain language materials	Recognize factors that contribute to making written health materials clear, understandable, and actionable	Read: “Reader-Friendly Materials” excerpt from Building Health Literate Organizations: A Guidebook to Achieving Organizational Change” (Abrams et al., 2014)	Assess patient-facing document using CDC Clear Communication Index (CDC, 2020); share findings and observations	-
	Identify tools to guide development and assessment of written health materials	Skim glossary (Agency for Healthcare Research and Quality, 2020); and Centers for Disease Control and Prevention (CDC) Clear Communication Index User Guide (CDC, 2019)
	Describe benefits and limitations of readability formulas for written health materials
	Describe how to use print health materials during patient encounters	Find short document for patients/families from a clinical setting
Shame-free care environment (welcoming; making it easy to ask questions) and organizational HL	Recognize features of a shame-free care environment	Read: Attributes of a Health Literate Organization (Brach et al., 2012); “The Care Environment” from Building Health Literate Organizations: A Guidebook to Achieving Organizational Change (Abrams et al., 2014)	Break-out group discussion and report-out on learning from web page assessments	Final quantitative self-assessmenta
	Describe the 10 Attributes of a Health Literate Health Care Organization			Final qualitative reflection
	Explain the Healthy People 2030 definitions of personal and organizational HL		Consider next steps to ensure their own ongoing use of HL practices and write them down
	Describe opportunities to recognize and respond to the need for using HL strategies to ensure high-quality, equitable, patient-centered care	Read Introduction and peruse HLE2: The Health Literacy Environment of Hospitals and Health Centers. Assess health entity web page using “Web Postings for Patients and Families” (Rudd et al., 2019)

Abbreviation: HL = health literacy.

^aSee Table A for Self-Assessment Questionnaire.

Table A.

Student Self-Assessment Questionnaire

Last Name: ___________________________ First Name: __________________________
Date: __________ Facilitator Name: ____________________________________________
Please help evaluate the curriculum by answering the following. Your responses will be
confidential and only shared in de-identified and summarized format.
1. On a scale from 1 to 10, how convinced are you of the importance of learning about health literacy and its relationship to patient care?
Not at all important									Very Important
1	2	3	4	5	6	7	8	9	10
2. On a scale from 1 to 10, how confident are you in your ability to:
- Define health literacy
Not at all confident									Very Confident
1	2	3	4	5	6	7	8	9	10
- Identify how health literacy may affect patients' health and well-being
Not at all confident									Very Confident
1	2	3	4	5	6	7	8	9	10
- Describe risk factors for or recognize signs that limited health literacy is affecting a patient
Not at all confident									Very Confident
1	2	3	4	5	6	7	8	9	10
- Use plain language routinely during patient and family encounters
Not at all confident									Very Confident
1	2	3	4	5	6	7	8	9	10
- Use teach-back routinely during patient and family encounters

Not at all confident													Very Confident
1	2	3				4	5	6		7	8	9	10
- Use principles for reader-friendly print materials
Not at all confident													Very Confident
1	2	3	4	5	6	7			8			9	10
3. Thinking about your encounters with patients and/or families, as Standardized Patients, in Longitudinal Practice, or elsewhere, please check all the elements of effective teach-back you have used more than half the time in the preceding month.
Use a caring tone of voice and attitude.
Display comfortable body language, make eye contact, and sit down.
Use plain language.
Ask the patient to explain, in their own words, what they were told.
Use non-shaming, open-ended questions.
Avoid asking questions that can be answered with a yes or no.
Take responsibility for making sure you were clear.
Explain and check again if the patient is unable to teach back.
Use reader-friendly print materials to support learning.
Document use of and patient's response to teach-back.
Include family members/caregivers if they were present.

Qualitative Analysis

The baseline reflection asked each student to submit a written reflection on the pre-work and session activities, discussion of how these influenced their perspectives, and description of a situation where they observed someone struggle with limited HL. The final reflection asked each student to submit a written description of at least one step they planned to take to continue use of HL practices.

Baseline reflections were completed by 473 consenting students (cohort 1–176; cohort 2–129; cohort 3–168); and final reflections by 432 students (cohort 1–155; cohort 2–112; cohort 3–165). Responses were input to Dedoose for data management and organization (Dedoose Version 9.0.17, 2021). A codebook was developed and applied by two trained qualitative research team members (EAA, JBV), using an inductive approach, allowing codes to emerge from the data during analysis, constant comparison to achieve precise code application, and memo writing (Corbin & Strauss, 1990). Coders collaboratively developed consensus codebooks for each reflection question. Codes and themes were discussed with the research team to ensure an iterative approach, interpret key findings, and promote reflexivity (Saldana, 2015). Saturation was achieved on key themes. After all responses for student cohort 1 were coded, unique, random samples of approximately 30% of responses in cohorts 2 and 3 were assessed for new themes that may have emerged related to cohort differences; saturation on key cohort 1 themes was confirmed, both within cohorts 2 and 3 themselves and among all cohorts together.

Quantitative Analysis

Among the 517 consenting students, 336 (65%) completed quantitative self-assessments across all three timepoints, providing longitudinal data for this analysis (493 baseline [489 complete], 423 complete midpoint, 407 complete final self-assessments). Statistical analysis of self-assessment items was conducted using R Studio (R Core Team, 2023). Baseline self-assessment data were collated to conduct an exploratory factor analysis (EFA)—a statistical technique that reduces the number of variables into common dimensions (factors). These explain two types of item variance: commonality (h2)—how much of the item variance is shared amongst the other variables; unique variance (u2)—variance unique to that variable (Field, 2013). A correlation matrix was used for this EFA. Pairwise deletion was used for missing data. The matrix met all necessary assumptions to be suitable for factor analysis.

EFA examination showed six latent variables with potential to serve as factors. Based on eigenvalues and scree plot inspection, a three factor EFA model was determined as the best fit (Cattell, 1966; Tabachnick et al., 2019; Velicer & Jackson, 1990). EFA was conducted using maximum likelihood extraction method with a Promax oblique rotation to allow for expected factor correlation (Costello & Osborne, 2005). Items were retained if they loaded onto a factor with coefficient of 0.30 or higher, accounting for at least 9% of item variance, providing evidence the item was essential (Bandalos & Gerstner, 2016).

After construct identification, a one-way repeated within-design analysis of variance (ANOVA) was implemented to identify significant factor differences across timepoints. All ANOVA assumptions were checked. Minor violations of normality and outlier assumptions were identified; this procedure has been shown to be robust to minor normality or outlier issues; a more conservative alpha level (0.01) was utilized to mitigate type 1 error risk (Blanca-Mena et al., 2023). ANOVA sphericity violation in construct 1 was rectified using the Greenhouse-Geisser epsilon correction (Field, 2013); this assumption was met for remaining factors.

Results

Qualitative Results

Three themes emerged, centering on emotions associated with a backward reading exercise (Table 1) (Weiss, 2007); shifts in awareness of HL as a challenge for patients; and behavior plans resulting from the curriculum.

Theme 1. The curriculum elicited student experiences and emotions they recognized as similar to those likely encountered by patients with limited HL.

Students reported various emotions during the backward reading exercise—feeling frustrated, confused, stressed, overwhelmed, or discouraged. Fewer felt angry, afraid, powerless, tired, worried, or worthless: “I also got frustrated about halfway into the exercise since I really wanted to read faster and understand better but could not.” Another explained frustration with limited time to complete the exercise, recognizing this as a potential barrier for patients.

Many found the exercise difficult and time-consuming: “I immediately felt overwhelmed by not being able to immediately recognize the words.” Students noted feeling embarrassment or self-consciousness. Some were surprised their reading level did not meet their internal standards. Others struggled when comparing their reading to classmates and related this to how patients may experience shame in front of health providers:

“I felt as though I could barely get past the first couple of words, while I noticed my peers only slightly struggling to read…I really began to empathize with patients who have poor health literacy as I felt embarrassed for not being able to understand like those around me.”

Some stopped reading, “gave up,” or did not want to read aloud or ask questions to avoid being seen going slower than peers.

Several said the exercise was particularly beneficial for medical students often used to succeeding academically and maybe never having experienced reading or comprehension challenges. Students commonly expressed a disconnect between reading and comprehending, some for the first time:

“Even though I eventually did manage to read the complete paragraph, however, I couldn't figure out the meaning of what I had read. Because I had to use a lot of mental power just to decipher each word, I wasn't trying to piece together the big-picture meaning while doing so.”

Students recognized their emotions during the exercise as those patients may feel in health care settings: “The feelings I experienced are definitely not ones which I would like to be feeling in a doctor's office.” They related these experiences to those of patients given health information they don't comprehend: “This was an emotional aspect to health literacy I had not been able to appreciate before.” Another noted a connection to their responsibility as future health providers:

“I realized this is a reality for many patients. Literacy isn't the only problem; understanding complex medical instructions involving medications, treatment options, therapies, etc., is not as simple as handing someone a set of discharge instructions. The exercise was enlightening…about how we as students and future providers need to ensure our patients are understanding what they are being told.”

When exploring data related to resistance or indifference to the exercise, a small number of responses expressed productive frustration. These were rooted in students (1) already knowing the material, (2) feeling the exercise was not a good comparison to actual experiences of patients with limited HL, and (3) expressing HL was incorrectly positioned as an individual concern rather than a structural challenge of a health care system that does not allow providers enough time with patients and encourages complex language in medical education and collegial networks.

Theme 2. Students expressed increased awareness of HL as a necessary area of focus and of its impact on patient health outcomes.

Students frequently characterized the experience as “eye-opening;” many had not known the high prevalence of limited HL. They reported learning not to assume patients' literacy levels and reported increased awareness of their own implicit bias regarding HL: “I should never assume that someone can read well just by talking to them.” Some commented they had previously only recognized limited HL as being associated with specific underrepresented populations, but after the exercise recognized it as a challenge that can affect anyone, regardless of language, immigration status, education, or employment:

“Coming from an immigrant family, I am biased to expect that limited health literacy is often the result of language barriers, and this was a good reminder that this is not necessarily the case.”

“It is easy to assume that someone with a job can read and understand information at a high school level. These exercises reminded me that even my ‘educated’ patients will probably not understand most of the information I give them.”

Students connected these insights to the value of universal HL precautions—using clear communication tools with everyone since anyone can experience limited HL depending on circumstances (Hadden & Kripalani, 2019): “It is better to ensure that a person is able to understand health-related documents rather than find out later that they are struggling.” They also recognized the impact HL may have on health outcomes and truly informed consent:

“I have learned that the stress that comes with not understanding can not only lead to patients not taking their medications correctly, but can also lead to them agreeing to procedures without understanding the risks because they are too embarrassed or intimidated to admit that they do not understand. I also never considered that patients might avoid seeking health care in the first place because something as simple as filling out a form can cause so much anxiety.”

Nearly unanimously, students connected their classroom experience to situations from their clinics or families' or friends' experiences with limited HL. They described working with people who had “no idea about their diagnosis,” “did not understand…the documents,” or “were unsure of exactly what medications the patient was supposed to be taking and at what frequency.” Many reported witnessing the beneficial impact of involving a family member or patient advocate, or positive experiences with preceptors or physician mentors who explained things slowly and took extra time with some patients. Seeing preceptors model good HL practices helped students feel more comfortable doing so themselves:

“...during one of my clinic sessions, there was a patient who was having a hard time understanding how the COVID-19 vaccines work.…My preceptor compared mRNA to a recipe that tells the cells what protein to make. He then used a piece of paper to draw out how the protein is recognized by immune cells and flagged for destruction. [My preceptor] asked the patient to explain the process back to him, and the patient seemed relieved to have a better understanding of the COVID-19 vaccine…”

Another shared a time they wished their preceptor had used Teach Back:

“I did not observe my preceptor attempting other methods of communication to check for understanding such as the Teach-Back method or simple diagrams. Maybe he had tried that in the past without success and the patient's mental health prohibited him from totally understanding. I would like to see him try the Teach-Back method when meeting with this patient in the future.”

Theme 3. Students indicated plans to apply HL practices in specific ways as a result of the curriculum.

Students described myriad ways they aimed to continue using HL skills. Table 2 shows categorized practices and representative student quotations.

Table 2.

Health Literacy Practices Students Described Planning to Apply

Health Literacy Practice	Student Quotations Representing Intent to Use Practice
Recognize signs patients are struggling to understand	“With every patient encounter, taking a moment to think about how the current patient may be affected by health literacy, regardless of their SES or my first impression of them.”
Change communication style	“The health literacy video and the discussions we had during our LG session talking about health literacy-related articles and experiences will help me be mindful of the tone, cadence and vocabulary I use when talking with patients and their families.”
	“It reminds me to be compassionate and slow down when speaking.”
Accept responsibility for patient understanding	“I will focus on doing a better job of saying ‘to make sure that I did a good job of explaining what we talked about.’”
Support patients in asking questions	“I will change how I ask for questions. Instead of saying, ‘Do you have any questions?’ I will say ‘What questions do you have?’ so that patients know that it's expected that they would have questions.”
Use open-ended questions	“One step I will take at this point is to ensure that paperwork and reading is not an issue through open-ended questions.”
Use plain language	“I will work on using plain language and familiarize myself with what words can be confusing for people and finding alternatives for that type of language to make it easier to understand.”
	“[I will] rehearse common medical phrases in a way that is patient-friendly, clear, and easy to understand.”
Check for understanding	“I will aim to chunk my summary into smaller amounts of information and check if they understand that before moving to the next snippet of information.”
	“If I give patients a print piece of material I want to go over it with them so that they understand it and check for understanding with clarification.”
Use Teach Back	“I would like to do at least one Teach-Back moment per day of patient encounters, and I will ask other members of the team (on rotations, etc.) to help remind me and hold me accountable.”
	“Intentionally choose to use the Teach-Back strategy and make a goal to do this in at least half of my patient encounters.”
	“I have an outline containing all of the questions that I ask during a standard patient interview. I will add a section regarding how the patient responded to Teach Back.”
Use reader-friendly materials	“I will make sure to ask the hospital/office that I am working at for reader-friendly printed materials to support patient learning.”
	“I want to use more visual devices like a whiteboard/diagrams/written lists tracking our discussion.”
	“I will look at patient handouts to identify if they are reader-friendly and try to mark them up to make things clearer for patients.”
Document health literacy practices	“I will document use of teach-back whenever I use it to hold myself accountable and understand patients' responses to Teach Back.”

Quantitative Results

Exploratory Factor Analysis

The three EFA factors identified from the 17 baseline self-assessment questionnaire items (Table B) were:

• A: Foundational factual and procedural knowledge (Foundations)

• B: Application of skills that create a shame-free environment (Shame-free Environment)

• C: Application of skills that effectively convey information (Conveying Information)

Table B.

Exploratory Factor Analysis of Baseline Self-assessment Items

Item Number	Item	Factor* loading			h 2	u 2
		A	B	C
1	Define health literacy.	0.75			0.53	0.47
2	Identify how health literacy may affect patients' health and well-being.	0.73			0.50	0.50
3	Describe risk factors for or recognize signs that limited health literacy is affecting a patient.	0.77			0.56	0.44
4	Use plain language routinely during patient and family encounters.	0.54			0.31	0.69
5	Use teach-back routinely during patient and family encounters.	0.70			0.56	0.44
6	Use principles for reader-friendly print materials.	0.68			0.51	0.49
7	Use a caring tone of voice and attitude.		0.41		0.16	0.84
8	Display comfortable body language, made eye contact, and sit down.		0.41		0.16	0.84
9	Use plain language.				0.11	0.89
10	Ask the patient to explain, in their own words, what they were told.		0.53		0.26	0.74
11	Use non-shaming, open-ended questions.		0.45		0.19	0.81
12	Avoid asking questions that can be answered with a yes or no.		0.42		0.18	0.82
13	Take responsibility for making sure you were clear.				0.11	0.89
14	Explain and check again if the patient is unable to teach back.		0.57		0.34	0.66
15	Use reader-friendly print materials to support learning.		0.37		0.14	0.86
16	Document use of and patient's response to teach-back.		0.63		0.35	0.65
17	Include family members/caregivers if they were present.				0.10	0.90

^*Factor A: Foundational factual and procedural knowledge (Foundations) (1 = not at all confident in ability to: 10 = very confident in ability to:) Factor B: Application of skills that create a shame-free environment (Shame-free environment) (Used more than half the time with patients and/or families in the preceding month^* – yes or no) Factor C: Application of skills that effectively convey information (Conveying information) (Used more than half the time with patients/families in preceding month^* – yes/no)

Questionnaire items 1 to 6 assessed recall of foundational factual and conceptual knowledge, and confidence in foundational procedures like plain language communication and Teach Back. These items formed the Foundations factor, with factor loadings from 0.54 to 0.75. The Foundations factor accounted for 17% of overall variance (eigenvalue 2.90).

Questionnaire items 7 to 17 were adapted from the Teach-Back Observation Tool (Abrams et al., 2023; Institute for Healthcare Advancement, 2024b) allowing students to self-report their recent performance of HL skills during self-assessment. Most items loaded on two factors describing conceptual HL skills subsets. Items loading onto the Shame-free Environment factor (items 7, 8, 11, 12) assessed application of skills like use of verbal and nonverbal signs of care and concern, and use of non-shaming, open-ended questions. Factor loadings ranged from 0.41 to 0.45. This factor accounted for 22% of overall variance with an eigenvalue (0.88) slightly below the recommended value of 1.00.

Items loading onto the Conveying Information factor (items 10, 14, 15, 16) assessed application of skills to effectively convey information and confirm understanding. Item loadings ranged from 0.37 to 0.63. This factor accounted for 7% of overall variance (eigenvalue 1.24).

Questionnaire items 9, 13, and 17 did not load convincingly with any factor and were removed from further analysis.

Repeated Analysis of Variance

Repeated measures of ANOVA compared self-assessment data across timepoints (baseline- midpoint-final) for each EFA factor. Table 3 shows self-assessment item and factor ratings across time points.

Table 3.

Composite Factor and Student Self-Assessment Item Ratings by Time Point

Item	Factor Name and Item	Baseline	Midpoint	Final	p
		Mean (SD)
Factor A: Foundational factual and procedural knowledge (Foundations) 1 = not at all confident in ability to - 10 = very confident in ability to)
		6.85 (1.25)	7.63 (1.17)	8.47 (1.00)	<.001
1	Define health literacy	7.06 (1.68)	7.76 (1.42)	8.72 (1.24)
2	Identify how health literacy may affect patients' health and well-being	7.74 (1.53)	8.41 (1.30)	8.97 (1.09)
3	Describe risk factors for or recognize signs that limited health literacy is affecting a patient	6.38 (1.74)	7.43 (1.46)	8.29 (1.36)
4	Use plain language routinely during patient and family encounters	7.15 (1.61)	7.75 (1.54)	8.26 (1.33)
5	Use Teach Back routinely during patient and family encounters	6.35 (1.92)	7.12 (1.81)	8.29 (1.46)
6	Use principles for reader-friendly print materials	6.41 (1.89)	7.28 (1.77)	8.29 (1.42)
Factor B: Application of skills that create a shame-free environment (Shame-Free Environment) (used more than half the time with patients and/or families in the preceding montha – yes or no)
		0.84 (0.18)	0.82 (0.21)	0.87 (0.19)	<.01
7	Use a caring tone of voice and attitude	0.99 (0.12)	0.97 (0.17)	0.98 (0.14)
8	Display comfortable body language, make eye contact, and sit down	0.95 (0.22)	0.90 (0.29)	0.94 (0.24)
11	Use non-shaming, open-ended questions	0.90 (0.30)	0.88 (0.33)	0.93 (0.26)
12	Avoid asking questions that can be answered with a yes or no	0.54 (0.50)	0.55 (0.50)	0.62 (0.49)
Factor C: Application of skills that effectively convey information (Conveying Information) (used more than half the time with patients/families in preceding montha– yes/no)
		0.12 (0.22)	0.25 (0.28)	0.38 (0.30)	<.001
10	Ask the patient to explain, in their own words, what they were told	0.14 (0.34)	0.38 (0.49)	0.58 (0.49)
14	Explain and check again if the patient is unable to teach back	0.15 (0.35)	0.33 (0.47)	0.52 (0.50)
15	Use reader-friendly print materials to support learning	0.15 (0.36)	0.20 (0.40)	0.28 (0.45)
16	Document use of and patient's response to Teach Back	0.06 (0.24)	0.10 (0.30)	0.15 (0.35)
No factor
		-	-	-	-
9	Use plain language	0.88 (0.33)	0.89 (0.31)	0.93 (0.26)
13	Take responsibility for making sure you were clear	0.66 (0.48)	0.77 (0.42)	0.84 (0.37)
17	Include family members/caregivers if they were present	0.57 (0.50)	0.62 (0.49)	0.71 (0.45)

^aComplete wording: Thinking about your encounters with patients and/or families, as Standardized Patients, in Longitudinal Practice, or elsewhere, please check all the elements of effective Teach Back you have used more than half the time in the preceding month.

Student self-assessment ratings for Foundations factor items increased significantly over the course [F(1.89, 632) = 297.71, p < .001]. Post-hoc analysis found significant rating increases from baseline to midpoint (p < .001) and midpoint to final (p < .001).

Self-assessment ratings for the Shame-free Environment factor increased modestly over the curriculum [F(2,670) = 5.22, p < .01], but not linearly. Ratings between baseline and midpoint varied non-significantly, then increased significantly from midpoint to final (p < .01).

Student self-assessment ratings for Conveying Information increased significantly throughout the course [F(2, 670) = 97.26, p < .001]. Ratings for this factor increased from baseline to midpoint (p < .001) and midpoint to final (p < .001).

Discussion

We developed a longitudinal HL curriculum and evaluation methods grounded in existing competencies and conceptual framework that elicited significant positive changes in medical student self-assessed knowledge, attitudes, and practices. To our knowledge, this is the first comprehensive qualitative and quantitative assessment of a longitudinal HL curriculum for undergraduate medical students. It demonstrates the value of a longitudinal approach, feasibility, and application of both individual and organizational HL perspectives to help consolidate learning.

Our study fulfills multiple features of a conceptual framework for HL training in health professions education, including sequenced, interactive sessions with multiple instructional modes; reflection; knowledge and skill education integrated within real-world or simulated settings; and varied assessments having validity evidence (Saunders et al., 2019). We provide quantitative self-assessment data to structure findings about students' HL knowledge and skills into three factors–Foundations, Shame-free Environment, Conveying Information (Abrams et al., 2023). These connect conceptually to meaningful domains within Bloom's Taxonomy: foundational factual and procedural knowledge; and HL skills application to create a shame-free environment and convey information (Anderson & Krathwohl, 2001). Quantitative results indicate most items contributed to a distinct factor. No item met the threshold for multiple factors, suggesting the assessment had three distinct domains. We found significant improvements in each domain over time. High baseline self-assessments in the Shame-free Environment factor limited the ability to detect substantial improvements over the curriculum.

We combined didactics with hands-on activities, interactive components, group discussion, and practice application opportunities (e.g., Standardized Patient encounters). We used publicly available online HL educational training resources, supported by student pre-work, ongoing reinforcement, and faculty teaching tips. Our findings support the importance of a longitudinal curriculum for undergraduate medical HL education (Coleman et al., 2022). A multifaceted, sequenced approach, with review before adding new content, and guidance on integrating HL skills during session activities likely contributed to progressive improvement in students' HL skills confidence.

Qualitative analysis demonstrated curriculum impact on students' insights into the ramifications of limited HL by promoting empathy and connecting classroom experiences to those encountered by patients with low HL. This was underscored by descriptions of frustration; recognizing the difference between decoding and comprehension; noting shame, embarrassment, and exhaustion when concealing lack of understanding; and acknowledging their own implicit biases.

Such responses relate to Bloom's knowledge dimension and increase students' likelihood of recognizing HL as a widely prevalent, often unidentified, problem associated with adverse health outcomes for individuals and health care entities. They support student comprehension and retention of content, application of HL skills to intervene, and valuation of ongoing HL skills use. Students expressed commitment and organization of values through specific visions of future accountability and advocacy they intend as they progress professionally. The first session led to higher-level insights on material not yet covered like the: need for HL universal precautions (describing not being able to tell whether someone is experiencing limited HL); identification of areas where HL is especially important, e.g., informed consent; and value of organizational HL (expressions of frustration that HL was incorrectly positioned as an individual concern rather than a structural challenge of the health care system); as well as recognition of the importance and benefit of faculty role-modeling.

Other strengths included analysis of three large diverse student cohorts over three academic years that encompassed shifts to virtual format during the COVID-19 pandemic, and implementation in a real-world environment with minor year-to-year variability in the overall LG curriculum, indicating robustness and flexibility of this HL curriculum.

Work in developing consensus-derived HL competencies has focused on observed or reported behaviors, identified as application of procedural knowledge in Bloom's Taxonomy. These procedural skills, like “avoids using medical jargon” or “routinely uses Teach Back” are essential and observable. They describe educational destinations without considering steps toward those destinations. Our study expands upon existing work by describing broader foundational facts and procedural knowledge that may be key to increasing HL skills application. Our study further refines classification of HL procedural skills into domains of skills that create shame-free and open communication environments, and skills with conveying information. Future work could contribute validity evidence for these domains and those we did not assess.

Limitations include lack of HL skills assessment by direct observation and inability to follow students beyond their second year of medical school, which is important as health professionals frequently over-report use of Teach Back and other HL skills (Feinberg et al., 2019). By comparing individuals to themselves over three timepoints, our study detects changes in self-reported behaviors rather than single timepoint self-assessments; this design attempts to mitigate the effects of overreporting. However, direct observation of students over time is needed to assess whether actual HL practices align with self-assessments and intentions. Attitudes and skills may have differed among students who did not opt to have their data included in the Repository. Since that occurred long before this study, it is unlikely to have systematically biased results. We did not conduct HL session-specific pre-/post-tests although relevant exams incorporating HL-related items performed well. Since previously reported medical school HL curricula, using various approaches, have consistently demonstrated at least short-term improvement on such assessments, we selected a different focus for our analysis (Maybury et al., 2024; Saunders et al., 2019). Finally, this is a single institution study, limiting generalizability.

We provide validity evidence for a comprehensive longitudinal medical school HL curriculum characterized by: periodic HL-specific content; ongoing reinforcement; interactive and hands-on learning strategies; and linking content with emotional and motivational perspectives like empathy for patients, the importance of improving outcomes, and primacy of safe, equitable, high-quality health care. After engaging in the 14-month HL curriculum, students had heightened HL awareness, self-reported increases in use of HL skills, and intent to carry HL practices into their future roles. Findings may be useful to educators, investigators, graduate institutions, health systems, and accreditation organizations to enhance HL education and training, develop and evaluate sustaining infrastructure, integrate into organizational HL efforts, and measurably improve health outcomes.

Funding Statement

Funding: This project was funded by a United States Department of Health and Human Services Office of Minority Health grant.