A Health Literate Evaluation of a Reproductive Health Education Program for Young Men with Sickle Cell Disease

Abstract

Background

Health literacy (HL) is known to be low among those with sickle cell disease (SCD). Reproductive health knowledge gaps exist among those with SCD, which drove the research team to create an education tool for adolescent males entitled, “Fertility edUcaTion to Understand ReproductivE health in Sickle cell disease” (FUTURES) in 2023. This study sought to evaluate and revise FUTURES using HL tools and a community-advised review to improve clarity, understandability, actionability, and relatability.

Methods

Four evaluators conducted a HL assessment of the original and revised FUTURES programs using the Patient Education Assessment Tool for Audio/Visual Materials (PEMAT-AV) and the modified Centers for Disease Control and Prevention Clear Communication Index (CDC-CCI-modified). Paired samples t-tests were used to compare PEMAT-AV and CDC-CCI-modified scores between the original and revised versions of FUTURES. Community advisory board feedback identified opportunities to improve program design.

Results

HL assessment revealed high HL demand of the original FUTURES program. The revised program had significantly higher understandability and clarity scores and was re-designed to be more engaging and representative of the intended audience. However, the CDC-CCI-modified score fell below the recommended threshold and actionability remained unchanged.

Conclusions

This study outlined an approach that prioritized HL and community voices when revising an education tool, which could be used in future efforts to design accessible and relatable patient education materials for this population and others. More work is needed to evaluate the impact of the HL revision process on patient knowledge and outcomes for this and other at-risk patient populations.

Introduction

Health literacy (HL) is defined as "the degree to which individuals have the ability to find, understand, and use information and services to inform health-related decisions and actions for themselves and others.” ¹ HL is vital to individuals’ health and wellness, as lower HL levels are associated with increased healthcare utilization, ² negative health behaviors, ³ and poorer health outcomes. ^4,5 However, 36% of Americans score at “basic” or “below basic” HL levels, ⁶ and Black/African American individuals often have lower HL levels than their White peers ^8,9 due, in part, to well-documented structural disparities in economic and educational opportunities, which can affect HL levels. ^7-9 These racial disparities in HL may perpetuate other health disparities^3,10 by disproportionately preventing those from underrepresented minority groups from navigating a complex healthcare system and making informed health-related decisions. ^11,12

To ensure all individuals can understand health information and make informed health decisions, the United States (US) Department of Health and Human Services created a National Action Plan to Improve Health Literacy in 2010. This Plan calls for collaborative efforts to create accurate, accessible, and actionable patient education materials that are understandable for individuals of all HL levels. ¹³ In response, HL scoring tools have been created to evaluate the HL-related demand of patient-facing materials, including the modified Centers for Disease Control and Prevention’s Clear Communication Index (CDC-CCI-modified) for written materials^14,15 and the Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-AV). ¹⁶ Increasing use of these tools has shown that many education materials available to patients and families exceed the HL levels of many Americans. ^17-22 These HL gaps are especially relevant for historically marginalized communities. For example, materials with higher HL demand may only be informative for those with higher HL levels. Those with limited HL may not be able to apply information discussed in these materials to their health-related decisions, contributing to existing health disparities, and perpetuating systemic disadvantages. ^2-6,10,23 Therefore, limiting HL demand should be prioritized when creating and revising patient education materials, particularly those for predominantly underrepresented minority groups who may have lower HL. ^24-27

SCD is an inherited blood disorder affecting approximately 100,000 U.S. individuals, most of whom are Black/African American. ^28,29 SCD is associated with multi-organ, lifelong complications resulting in frequent hospitalizations and acute care visits. ^30-32 Fortunately, care for SCD is improving, and children with SCD are living well beyond their reproductive years, ^33,34 making education about reproductive issues for adolescents with SCD more salient. ^35-41 This shift has also highlighted that SCD-specific concerns about reproductive health are often unmet. ^37,39,42-45 This may be related to limited patient education on SCD-specific reproductive health topics, ^27,37,43,46 but also the discrepancy between the HL level of many adolescents with SCD and their caregivers and the HL demands of existing materials. ²⁰ An estimated 13% of the SCD population in the US immigrated from the Caribbean, Western, Middle and Southern Africa, and immigrants are more likely to have lower HL. ^47-49 Additionally, despite demonstrating high levels of interest in health-related information, only 23% of 9^th graders had adequate HL levels with 44% showing health-related reading skills at or below a 7^th grade level. ⁵⁰ Considering the lifelong impact of SCD, ^30,31 how common limited HL is among this community, ^24-27 and the added complexity of SCD’s impact on reproductive health (i.e., genetic transmission and fertility), ^38,40,41 clear and understandable materials addressing these topics are necessary to promote informed decision-making for those with SCD.

To bridge reproductive health education gaps, our team collaborated with a community advisory board (CAB) of young men with SCD and caregivers of adolescents with SCD to create a novel education program in 2023 entitled, “Fertility edUcation To Understand ReproductivE health in Sickle cell disease” (FUTURES). FUTURES is an interactive, self-guided, digital education program discussing reproductive health in the context of SCD and its treatments for adolescent males with SCD (ages 13-21) and their caregivers. ^27,51,52 FUTURES consists of 41 slides separated into two modules. It includes written text accompanied by verbal narration taking 10-15 minutes to complete. ^51,53 Multiple choice and true/false knowledge check questions are incorporated throughout both modules to maintain engagement and promote knowledge gain. ⁵¹ Despite demonstrating promise as a pilot, knowledge gain was limited among participants with limited HL, ²⁷ suggesting FUTURES’ HL demands may exceed the HL levels of the intended audience and highlighting an opportunity to improve the program. Prior studies have used HL scoring tools combined with input from community members to increase knowledge gain from and acceptability of education materials for people with SCD and sickle cell trait (SCT). ^54,55 As such, the aims of this study were to: 1) use validated HL scoring tools in conjunction with a community-advised approach to identify opportunities to improve the FUTURES program and 2) use these results to improve FUTURES’ clarity, understandability, actionability, relatability, and design.

Methods

HL Evaluation Team and Community Advisory Board

The HL evaluation team was comprised of four clinical research coordinators (MG, JW, SL, and ZC), two of whom were involved in the initial program development and pilot testing. ^27,51 Prior to evaluation, the team watched the American Medical Association HL video, ⁵⁶ attended an interactive lecture developed by a national HL expert (MAA) that included practical strategies for improving written and audiovisual programs, completed the Always Use Teach-back! Toolkit learning module, ⁵⁷ and reviewed the PEMAT-AV and CDC-CCI user guides. ^58,59

A CAB was established when FUTURES was initially developed; ²⁷ two of the original members and four new members were invited to provide feedback on the program revisions. CAB participation was limited to an advisory role on FUTURES and no personal health information was collected (thus deemed to fall in an exempt category of research by the Institutional Review Board). CAB members were invited based on their familiarity with FUTURES either through participation on the CAB that developed the initial version of the program or as participants during pilot testing. ²⁷ CAB members included four male youth with SCD and two mothers of adolescent males with SCD, reflecting the intended study population.

Evaluation Tools

The CDC-CCI-modified^14,15 and the PEMAT-AV¹⁶ are research-based HL tools measuring the understandability, actionability, and clarity of patient education materials. These tools were selected for this evaluation due to their use across similar programs and studies. ^22,54,55

The CDC-CCI-modified is comprised of 13 yes/no questions, each worth one point, and four free response questions. ^14,15 It has four subscales: core items, behavioral recommendations, numbers, and risk, as well as a composite score to assess clarity and comprehensibility. ¹⁴ Clarity is defined as the degree to which the material effectively conveys the main message and risks of behaviors associated with the topic. ⁵⁹ The CDC-CCI-modified uses ratings on individual questions to calculate a composite score ranging from 0-100% with higher scores indicating the material is clearer and more understandable. Per the scoring guidelines, a composite score >90% is considered acceptable. ¹⁴

The PEMAT-AV is comprised of 17 agree/disagree questions, each worth one point, split into two subscales: understandability and actionability. ¹⁶ Understandability is defined as the degree to which people with varying HL can explain main messages. ¹⁶ Actionability is defined as the degree to which people with limited HL can identify the appropriate next steps based on information provided in the material. ¹⁶ Ratings on individual PEMAT-AV questions were used to calculate a score that could range from 0-100%; >70% on the PEMAT-AV is considered acceptable. ^14,16 Higher scores on the PEMAT-AV indicate that the material is more understandable and actionable.

Evaluation and Revision Process

From August 2023 – September 2023, HL evaluators independently scored each of FUTURES’ 41 slides and the voiceover using the HL tools. Since each slide was not designed to be viewed in isolation, the program was also scored as a whole to account for 1) the context provided by viewing the complete program and 2) the possibility that individual slides may have scored poorly independently while still contributing to the overall clarity, understandability, and actionability of the program. ⁵⁴

Evaluators met four times throughout the process to discuss their scores, resolve discrepancies, agree on a final score for each slide and the program overall, and suggest and draft revisions for FUTURES. Strong inter-rater reliability was demonstrated when scoring using the PEMAT-AV (κ=0.92) and the CDC-CCI-modified (κ=0.87). ⁶⁰ Evaluators used low PEMAT-AV and CDC-modified scores on the original version of FUTURES to identify areas for improvement and drafted revisions to the program to reduce HL demand. Following creation of a draft of the revised program, evaluators began an iterative, community-advised revision process. Four meetings were convened to accommodate CAB members’ schedules so that each CAB member could attend one of these meetings with multiple research team and CAB members. During these meetings, CAB members viewed drafts of the revised FUTURES slides in comparison to the original slides and provided feedback on the language and illustrations that were used. Their feedback was incorporated to make additional changes before finalizing the revised program. Finally, the HL evaluation team rescored the revised program using the same scoring procedure.

Analytic Plan

Analyses were completed in SPSS. Descriptive statistics were used to assess the CDC-CCI-modified and PEMAT-AV scores for each individual slide and the program as a whole for the original FUTURES program and the revised FUTURES program. Paired samples t-tests for each individual slide and the overall scores for both tools (i.e., CDC-CCI-modified and PEMAT-AV) assessed differences between the original and revised version of FUTURES. Because the revision process involved condensing and modifying the content, the revised program was shorter than the original the program. Thus, two evaluators paired the slides, based on similarity of content, from the original and revised versions of FUTURES for analyses.

Results

HL Evaluation

CDC-CCI-modified composite scores on all individual slides (n=41) in the original FUTURES program fell below 50%. Following revision, composite and subscale scores on individual slides significantly increased. Paired samples t-tests demonstrated a significantly higher mean CDC-CCI-modified composite score of the revised program’s individual slides (M = 35.68%, SD = 15.65%) compared to the original version (M = 28.19%, SD = 14.32%), t(33) = 3.11, p = <.01. Additionally, significantly higher scores were demonstrated on the CDC-CCI-modified core and risk subscales, across individual slides (Table 1). When scored as a whole program, FUTURES original- and revised-composite raw scores were 58% and 75%, respectively.

Table 1.

Paired samples t-test Results Comparing the Means of the CDC-CCI-modified and PEMAT-AV Scores on the Individual Slides of the Original and Revised version of FUTURES.

Tool	Subscale	Version	Mean (%)	SD (%)	t(33)	p
CDC-CCI-modified	Core	Original	31.81	15.7	4.21	<.01
		Revised	44.55	17.52
	Behavioral Recommendations	Original	4.41	14.40	2.03	.051
		Revised	14.71	33.78
	Numbers	Original	5.88	16.35	n/a	n/a
		Revised	5.88	16.35
	Risk	Original	16.18	36.35	2.10	.04
		Revised	4.41	18.94
	Composite	Original	28.19	14.32	3.11	<.01
		Revised	35.68	15.65
PEMAT-AV	Understandability	Original	77.66	9.74	4.77	<.01
		Revised	87.54	12.26
	Actionability	Original	8.82	23.65	1.49	.15
		Revised	15.69	28.71
	Composite	Original	60.07	10.19	3.81	<.01
		Revised	67.68	11.46

Note. The CDC-CCI-modified subscale, Numbers, did not have any variation and thus no test statistics are available.

The individual slides from the original FUTURES program received PEMAT-AV understandability scores that ranged from 56% - 100%. Composite and understandability scores across individual slides improved significantly following the revision of FUTURES. The mean understandability score on the individual slides in the revised version was significantly higher (M = 87.54% SD = 12.26%), than the mean understandability score on the individual slides in the original version (M = 77.66% SD = 9.74%), t(33) = 4.77, p = <.01. When scored individually, only 6% (n = 2) of slides in the original version scored >50% on the actionability subscale, whereas 21% (n = 7) scored >50% after revision. Scored as a whole program, FUTURES received a raw score of 67% on both understandability and actionability subscales, prior to revision. Although the understandability score rose to 83% following revision, the actionability score remained the same.

Program Revisions

The HL evaluation team identified the following areas for revision: clarifying FUTURES’ main message, revising the visuals to improve clarity, clarifying the calls to action, and defining more of the medical terms. To focus the main messages, the research team narrowed the content on genetic transmittance of SCD by combining key takeaways from two slides into one (Figure 1). Additionally, the research team streamlined content about the impacts of SCD on fertility and simplified diagrams to improve the clarity and purpose of the material (Figure 1). Due to low scores on the actionability subscale, the content on contraception and encouraging their partners to be tested for sickle cell trait was revised to directly address the audience using active voice. On both the PEMAT-AV and the CDC-CCI-modified, the original slides lacked definitions of medical terms (i.e., fertility, reproductive health); plain language definitions were added at their first instance of use.

Figure 1.

FUTURES Program Revisions and Sample Scores.

CAB member feedback included: varying hair styles and skin tones of the illustrations to be more representative of the diverse SCD population (Figure 1), providing insight into medical terms that did not need to be defined because they were well understood in the SCD community (e.g., hemoglobin), and modifying terms used to address the intended audience (e.g., adolescents vs. young men). Additionally, CAB members suggested changing FUTURES’ narration to be more relatable and engaging for the intended audience. Thus, a minority male resident physician (AP), who has provided care for youth with SCD, narrated the revised program.

Discussion

These findings highlight that the HL demand of FUTURES exceeded recommended threshold scores on two validated HL scales and, likely, the HL levels of many in the SCD community. ^14,16,24-27 Notably, the understandability and clarity of this program were significantly improved through targeted revisions informed by a HL assessment and community-advised review. Furthermore, the program’s visual and audio components were updated to better represent the diversity of the SCD community with the goal of providing a more engaging experience.

This study adds more evidence that existing education tools about SCD may be inaccessible because of low understandability and clarity. ²⁰ This discrepancy between HL demands of patient education materials and the limited HL levels of many patients with SCD and their families^24-27 may diminish the educational value of health-related materials. ^27,61 Furthermore, using materials with high HL demand prevents those with limited HL levels from gaining the knowledge needed to make informed decisions. ^27,61 Specifically, this may limit the understanding and uptake of fertility testing for adolescent males with SCD. These findings emphasize the urgency of considering HL level when creating patient education materials to promote equitable access to health-related information. ²³ Our findings also highlight that using validated HL scales to assess existing education tools and to guide their revision can improve their understandability and clarity, and presumably improve the educational impact they have on patients and families. Similar outcomes have been shown when creating accessible education materials for people with SCD and SCT^54,55 suggesting that efforts to reduce HL demand of health-related content for SCD can result in tools that are more useful for the community.

Although understandability and clarity were improved in the revised version of FUTURES, there is room to improve this program’s actionability, as it remained unchanged. This may be because the FUTURES program was created to provide introductory knowledge about potential SCD-related risks to reproductive health rather than to drive action⁵¹ given reproductive health knowledge gaps among adolescents with SCD. ^27,37,43,46 In addition, the CAB sessions focused more on ensuring content was understandable and representative rather than maximizing actionability. Despite this, there were some aspects of the program designed to encourage action from viewers, such as obtaining partner testing for SCT, using contraception, and discussing questions with healthcare providers. Actionable materials will be needed once education about reproductive health in SCD becomes more widespread and as more evidence-based guidelines become available to inform the most appropriate actions to take to mitigate SCD-related reproductive health risks.

While the overall scores increased after program revisions and the revised version met the threshold score on the PEMAT-AV, neither the original nor revised versions met the thresholds for acceptable scores on the CDC-CCI-modified. ^14,16 While this suggests that more revisions may be needed, the score improvement and overall final scores seen in this study were greater than those seen in a similar effort to improve educational slide sets for a behavioral intervention on sugary beverages which used the original CDC-CCI tool. ²² Furthermore, compared to 13 SCD-related patient education materials evaluated with the PEMAT-AV and original CDC-CCI, the revised FUTURES program outscored all 13 materials on the PEMAT-AV and scored at or above 12 out of 13 materials on the CDC-CCI. ²⁰ Regardless, the below threshold scores seen here may have occurred for several reasons. First, the CDC-CCI-modified score was lower than the threshold primarily because most of the behavioral recommendations that were provided in FUTURES did not include a risk and benefit discussion, resulting in missed points in the evaluation process. This was done to minimize the time required to complete the program, particularly since the recommended behaviors had minimal inherent risks. Although the PEMAT-AV score surpassed the threshold, there remain opportunities to further improve the PEMAT-AV score. For example, the PEMAT-AV assessment primarily identified the limited actionability of the program. In addition, the PEMAT-AV score was limited because FUTURES incorporates knowledge checks throughout to maintain engagement rather than formal summaries, which are specified content on the PEMAT-AV. Descriptive headers throughout are also recommended, but headers were mainly used to note the beginning of each module in FUTURES to minimize the amount of text on slides, which resulted in missed points on the PEMAT-AV. ¹⁶ Even still, it is important to note that the suggested threshold scores have not been rigorously tested to see if achieving them is required to attain high knowledge among individuals of all HL levels or if programs that score below these thresholds may still effectively educate and enable informed decision-making. Improved scores, even if not reaching suggested thresholds, represent progress toward addressing the HL-related demands of the materials. Furthermore, the HL review process necessitates continued review and revision alongside intended audiences. Thus, future iterations of FUTURES can prioritize the areas identified here. Future HL research is needed to explore and empirically test appropriate threshold scores indicating maximized knowledge gain and informed decision-making.

In addition to the HL assessment, this study incorporated community voices in the revision process to improve program design and ensure terms were adequately defined. Previous work has highlighted the importance of community-advised approaches to clinical research in SCD to bridge gaps in trust and ensure the patient experience is heard and understood. ^51,62,63 In addition to designing materials that are easy to understand and act on, including the populations served throughout the design, implementation, and evaluation of health information and services is also an important attribute of health literate organizations. ⁶⁴ The collaborative approach outlined here incorporated ongoing discussions between the study team and community members using an iterative revision process. This allowed the study team to improve depictions of individuals with SCD to better represent SCD community diversity, create a more engaging design throughout the program, and minimize redundancy of defining terms with which the SCD community is already familiar. This investigation also builds on previous work to further illustrate that community-driven development of education tools can create quality and relatable products. ^54,55

There are multiple potential implications of the revised FUTURES program and the approach used to conduct a HL-informed revision. First, the use of the CDC CCI-modified and the PEMAT-AV resulted in a clearer and more understandable FUTURES program which may improve the educational impact of this tool. Given the mismatch between the HL levels of the population and the HL demand of existing patient education materials, ^65,66 HL informed revisions, such as the one demonstrated in this study, are a necessary step towards promoting health equity. By creating materials that are clear and inclusive of a variety of HL levels, more people will be able to understand and act on important health information. ⁶⁷ Second, findings from this study also advance the evidence and application of the CDC CCI-modified and the PEMAT-AV specifically for their use in guiding the revision of interactive patient education tools with both audio and visual components. Increasing the use of audio and visual components in health communication and education calls for tools that can be used to optimize these to achieve their purpose. Third, though this study was conducted within the context of the FUTURES program, the process of using the CDC-CCI-modified and the PEMAT-AV could be expanded to assess and improve the clarity, understandability, and actionability of other patient education materials. This adds further evidence of the utility of applying these tools to patient education materials to promote safe, equitable, and high-quality health care. Finally, future research using similar community-advised approaches may improve the impact of interactive, health-related education programs and increase the SCD educational content in general.

These findings should be considered within the context of several limitations. First, this study was performed at a single site in the midwestern United States, which may limit the generalizability of these findings. In addition, this study did not assess the impact of the revised program on reproductive health knowledge. Future multi-center studies are needed to evaluate the impact and acceptability of the revised program on a larger sample of young adult males and their caregivers. Third, this study only included English-speaking individuals and female caregivers in the CAB. As limited HL is common among non-English speakers, ⁶ future research is needed to evaluate accessibility of this content in non-English-speaking individuals and to include the male caregiver perspective. Fourth, the HL assessment tools used in this study were designed for adult populations but have been used for other pediatric populations. ⁶⁸ Even still, ensuring that the HL demand of education materials is low may be particularly important for adolescents given that this population may have lower HL. ⁵⁰ Finally, similar to other studies involving CABs, this study only reflects the perspectives of a small number of community members. ^69,70

Conclusions

This study found that HL-focused revisions informed by validated HL assessment scales and a community-advised review improved the clarity and understandability of an interactive, multi-media education program for young adult males with SCD and their caregivers. Although improved, the final overall score of the revised program remained below the recommended threshold on one of two HL scales used, suggesting there is still room to improve the actionability and organization of this information. Furthermore, this study outlined an approach that prioritized both HL and community voices when revising an existing education tool, which could be used in future efforts to design accessible patient education materials for this population and others. Additionally, the findings from this study advance the applications of the CDC-CCI-modified and the PEMAT-AV to interactive patient education materials with both audio and visual components to promote high-quality health education and healthcare. HL-informed revisions, such as the one demonstrated in this study, are a necessary step towards addressing the goals of the National Action Plan to Improve Health Literacy by reducing the HL demand of existing patient education tools and promoting health equity.¹³ Future research is needed to further evaluate the impact of this HL evaluation approach on understanding and clinical outcomes for this and other at-risk patient populations.

Biographies

Megan M. Griffith is a research coordinator in the Center for Biobehavioral Health at Nationwide Children’s Hospital.

Joseph Walden is a research coordinator in the Center for Biobehavioral Health at Nationwide Children’s Hospital.

Sophia Liles is a research coordinator in the Center for Biobehavioral Health at Nationwide Children’s Hospital.

Zachary Colton is a research coordinator in the Center for Biobehavioral Health at Nationwide Children’s Hospital.

Mary Ann Abrams is a Clinical Assistant Professor and Longitudinal Group Associate Program Director at The Ohio State University College of Medicine overseeing work in health literacy.

Ben McCorkle is an Associate Professor of English at The Ohio State University.

Albert Phillips is a pediatrics resident at Nationwide Children’s Hospital.

Toyetta Barnard-Kirk is a community partner.

Leena Nahata is a pediatric endocrinologist at Nationwide Children’s Hospital and Professor of Clinical Pediatrics at The Ohio State University College of Medicine.

Susan E. Creary is a pediatric hematologist at Nationwide Children’s Hospital and an Associate Professor of Pediatrics at The Ohio State College of Medicine.