Addressing health literacy in patient decision aids: an update from the International Patient Decision Aid Standards

Abstract

Background:

There is increasing recognition of the importance of addressing health literacy in patient decision aid (PtDA) development.

Purpose:

An updated review as part of IPDAS 2.0 examined the extent to which PtDAs are designed to meet the needs of low health literacy/disadvantaged populations.

Data Sources:

Reference list of Cochrane review of randomised controlled trials (RCTs) of PtDAs (2014, 2017 and upcoming 2021 versions).

Study Selection:

RCTs that assessed the impact of PtDAs on low health literacy or other disadvantaged groups (i.e. ≥50% participants from disadvantaged groups and/or subgroup analysis in disadvantaged group/s).

Data Extraction:

Two researchers independently extracted data into a standardized form including PtDA development and evaluation details. We searched online repositories and emailed authors to access PtDAs to verify reading level, understandability and actionability.

Data Synthesis:

Twenty-five out of 213 RCTs met inclusion criteria illustrating that only 12% of studies addressed the needs of low health literacy or other disadvantaged groups. Reading age was calculated in 8/25 studies (33%), which is recommended in previous IPDAS guidelines. We accessed and independently assessed 11 PtDAs. None were written at 6^th grade level or below. Ten PtDAs met the recommended threshold for understandability, but only 5 met the recommended threshold for actionability. We also conducted a post-hoc subgroup meta-analysis and found that knowledge improvements after receiving a PtDA were greater in studies that reported using strategies to reduce cognitive demand in PtDA development compared to studies that did not (Chi²=14.11, p=0.0002, I²=92.9%).

Limitations:

We were unable to access 13 out of 24 PtDAs.

Conclusions:

Greater attention to health literacy and disadvantaged populations is needed in the field of PtDAs to ensure equity in decision support.

INTRODUCTION

Health literacy is defined as people’s knowledge, motivation and competences to access, understand, appraise, and apply health information in order to make judgments and decisions, and act upon their decision in everyday life concerning healthcare, disease prevention and health promotion.¹ Low health literacy is prevalent in many countries including those with high incomes; estimates of basic or inadequate literacy range between 1/3 and ½ of the population depending on the measure used in countries such as US, UK and Australia.^2–5 Patient decision aids (PtDAs) are evidence-based tools designed to help patients make thoughtful choices relating to healthcare options,⁶ but few address the needs of adults with lower health literacy. In a 2013 review of 97 trials, only three PtDAs overtly addressed the needs of lower health literacy users.⁷ In 90% of trials, user health literacy and readability of the PtDAs were not reported.⁷

There has been increasing international recognition of the importance of addressing the needs of patients and consumers with low health literacy when developing shared decision-making tools.⁸ The original International Patient Decision Aid Standards (IPDAS) quality criteria checklist, for example, included three criteria pertaining to the use of ‘plain language’ (see Box 1). Although this was reduced to one item in the 2009 (IPDASi v3.0) and 2013 (IPDASi v4.0) revisions of the standards (“The decision support technology (or associated documentation) reports readability levels (using one or more of the available scales)”) several shared decision-making tools designed since then have been guided by this criterion.

Box 1. International Patient Decision Aid Standards (IPDAS).

Criteria Pertaining to the Use of “Plain Language”

IPDAS quality criteria checklist (2005)

• Is written at a level that can be understood by the majority of patients in the target group

• Is written at a grade 8 equivalent or less according to readability score (SMOG or FRY)

• Provides ways to help patients understand information other than reading (audio, video, in-person discussion)

IPDAS instrument (IPDAS v3.0) (2009) and IPDAS minimal criteria (IPDASi v4.0) (2013)

• The decision support technology reports readability levels (using 1 or more of the available scales)

Evidence reviews show that the comprehension of health information among individuals with low health literacy can be improved through modifications to communication and other mixed-strategy interventions.^9,10 For example, two systematic reviews (and the individual projects that they draw upon) highlight benefits of reducing medical jargon and presenting essential information only (or first) in materials for adults with lower health literacy, as well as the potential utility of different communication formats (e.g., illustrated text; spoken animations; pictorial information).^9,10 There has also been growing movement towards, and evidence-base for, consumer involvement in the development and review of patient information materials.¹¹ However, it is unclear the extent to which new evidence and recommendations related to health literacy have been implemented in the development and evaluation of PtDAs since 2013.

Aims

This systematic review is an update of our previous 2013 review of the evidence relating to health literacy and PtDAs for IPDAS 2.0. We aimed to examine the extent to which PtDAs had been designed and tested with lower health literacy and disadvantaged populations.

Review questions

We reviewed literature on PtDAs to understand the extent to which they have been tailored to low health literacy and other disadvantaged groups. Our research questions are summarized below:

1. What proportion of PtDAs have considered low health literacy and other disadvantaged groups in their development and/or evaluation?

2. How have these PtDAs been tailored for adults with low health literacy and other disadvantaged groups?

3. What are the readability and cognitive demand scores of the PtDAs tailored for adults with low health literacy and other disadvantaged groups?

METHODS

Protocol and registration

The aims and methods for this systematic review were registered on PROSPERO (Registration number CRD42019159042). Reporting is guided by the Preferred Reporting Items for Systematic Reviews (PRISMA) checklist.¹²

Information sources and search strategy

We searched for published trials of PtDAs in the 2014, 2017 and upcoming 2021 versions of the Cochrane systematic review of randomized controlled trials (RCTs) of PtDAs.¹ This series of Cochrane reviews aim to examine the effects of PtDAs for individuals who are making treatment or screening decisions (aged 18+, making decisions for themselves, a child, or an incapacitated significant other). Their scope covers all published RCTs that compare the effects of PtDAs on decision-making outcomes (e.g., knowledge, risk perceptions and involvement in decision making) relative to some alternative ‘usual care’ (e.g., general information, clinical practice guideline, placebo, no intervention).^6,13 PtDAs are defined as “evidence-based tools designed to help patients make specific and deliberated choices among healthcare options” (p. 7).⁶

Eligibility criteria

To be included in the current systematic review, studies had to include at least 50% of participants from disadvantaged groups and/or conduct a separate analysis exploring the impact of the PtDA on disadvantaged groups. This criteria was adopted in order to be more inclusive of studies with disadvantaged populations, not only lower health literacy, due to the known correlation between health literacy and socially-defined factors (such as educational attainment; socioeconomic status).^3,14 Disadvantaged groups were defined using the following eight criteria, based on previous work by Durand et al:¹⁵

1. People who have lower literacy and/or lower health literacy

2. People who have lower educational attainment

3. People who are socially disadvantaged with respect to poverty or lower socioeconomic status

4. People who are socially disadvantaged as a result of their ethnicity or race

5. People who are socially disadvantaged with respect to geographical location (areas described as disadvantaged/or medically underserved)

6. People who are uninsured or on public health insurance

7. People who have lower numeracy

8. People who are socially disadvantaged as a result of speaking a primary language that differs from the official language(s) of their country of residence.

All conditions and clinical settings (e.g., lay care, primary care, secondary/tertiary care) were included.

Screening and study selection

Articles were independently screened in two stages by JS and OM as follows: screening of titles and abstracts followed by the retrieval and screening of relevant full-text articles using the inclusion criteria described above. Disagreements were resolved through discussion, or with the help of a third reviewer (DM).

Data extraction, synthesis and analysis

We used a standardized form to extract relevant data, which was pilot tested and iteratively revised. Of nine researchers (DM, JS, OM, KM, TC, AG, AL, AH, SS), two were assigned to each study to independently extract data including the methodology of each study and details of the development and/or evaluation of the PtDAs. To access PtDAs not published with their corresponding articles, we searched the Decision Aid Inventory Library maintained by the Ottawa Hospital Research Institute.¹⁶ If the PtDA was not publicly available, we made two email attempts via the corresponding authors’ emails specified on the manuscript.

Data items extracted and synthesis details relating to each research question are presented in Table 1.

Table 1.

Data items for each research question.

Research question	Data items, summary measures and synthesis approach
1a . How many PtDAs considered low health literacy and other disadvantaged groups?	Proportion of PtDAs included in the 2014, 2017 and upcoming 2021 versions of the Cochrane systematic review that consider low literacy and other disadvantaged groups in their development or evaluation as stated in the manuscript or determined by study participants or analyses.
1b . How have these PtDAs been tailored for adults with low health literacy and other disadvantaged groups?	Narrative description of how PtDAs meeting our inclusion criteria were developed or tailored for low literacy and other disadvantaged groups, including an inventory of the strategies used and a summary of data recording the acceptability and attitudes of PtDAs among the target audience.
1c. What are the readability and cognitive demand scores of the PtDAs tailored for adults with low health literacy and other disadvantaged groups?	We reported the proportion of PtDAs that were written at a ≤8th grade level and < 6th grade. Grade reading level was calculated using the Online-Utility.org readability calculator.18 We average the SMOG and Gunning-Fog scores to provide an overall grade reading level for PtDAs we accessed. Video format PtDAs were transcribed verbatim to assess the readability of the transcript. Full text of the PtDAs were prepared in line with guidelines by removing text that is not in full sentences (i.e. titles, headings, subheadings, short captions), embedded punctuation, and document design elements (e.g., gaps, white spaces, pictures and images, and text boxes). Bulleted text was included if it was in a full sentence or could be adapted to form a full sentence (e.g., moving words from the stem into the list) and punctuated by adding a full stop. Footnotes were removed unless they were essential to understanding the main PtDA content. We reported the proportion of PtDAs that met the PEMAT criteria for ≥70% actionability and understandability. Understandability and actionability were calculated for each PtDA that we are able to access using the Patient Education Materials Assessment Tool (PEMAT; see Box 2).19,20 PEMAT scores were independently scored by two team members (JS and OM) and disagreements were resolved via discussion with a third.

Risk of bias assessment

Risk of bias assessments were not conducted for studies that had already been assessed as part of the Cochrane Review (2014 and 2017 versions). Recent studies that had not yet been assessed for the upcoming 2021 version of the Cochrane Review were assessed for risk of bias by two independent reviewers from the study team using the revised Cochrane tool.¹⁷

Subgroup and sensitivity analysis

No subgroup or sensitivity analyses were pre-specified. However, after a concurrent meta-analysis and narrative synthesis exploring the impacts of PtDAs designed for low health literacy and other disadvantaged groups found evidence of improvements in decision-making outcomes (Yen et al., in this issue), we decided to conduct a subgroup meta-analysis to further investigate the impacts of tailoring strategies on knowledge and decisional conflict. Specifically, we examined whether these key decision-making outcomes differed in studies that reported assessing readability or using strategies to reduce cognitive demand compared to studies that did not report such strategies. The methodologies used in this meta-analysis and subgroup meta-analysis are described in full in our companion paper (Yen et al., in this issue).

RESULTS

Study characteristics

We analyzed 25 articles that met the inclusion criteria as they included at least 50% of participants from a disadvantaged population (n=10 studies), conducted a separate analysis including participants from a disadvantaged population (n=5), or did both (n=10; Table 2). (Figure 1). The total number of participants in all included trials was 9865.

Table 2.

Characteristics of included studies (total participants in all trials = 9,865)

			Inclusion criteria
Authors and year	Area and Country	Sample size for baseline characteristics	Poverty or SES	Ethnicity or race	Education	Low literacy/health literacy	Geographical location	Uninsured or public health insured	Lower numeracy	Different language	Separate analysis	Context of decision	Target population Participants targeted for recruitment, as described by authors	Basis on which population was classified to be within a disadvantaged group
Boulware et al 2018	Baltimore MD, USA	92	✓	✓	✓			✓				Living donor kidney transplantation	Patients who initiated hemodialysis within 2 years of screening, AA, English speaking, 18+ years	59% ≤$20k income, 100% AA, 73% ≤high school, 74% no private insurance
Brenner et al 2016	Nth Carolina, New Mexico USA	262		✓				✓		✓	✓	CRC screening	50–75 years, English or Spanish speaking, average risk, not up to date with recommended screening, had upcoming appointment	61% Lat., 88% no private insurance. Separate analysis to test moderating effects of race, insurance, education, literacy, language and site
Diefenbach et al 2018	USA	349									✓	PCa treatment	Patients with PCa (localized), not yet made a treatment decision, English-speaking, access to computer	Separate analysis to test moderating effects of race and education
Hoffman et al 2017	Houston USA	88		✓							✓	CRC screening	49–75 years, AA, due for screening, English speaking	100% AA. Separate analysis to test moderating effects of health literacy
Ibrahim et al 2017	Philadelphia USA	336	✓	✓								Total knee replacement	50+ years, AA, chronic and frequent knee pain, evidence of knee osteoarthritis	50% annual household income <$15k, 100% AA
Jimenez et al 2017 (pilot)	Philadelphia USA	64 (parent-child dyads)		✓							✓	Early intervention for developmental concerns in children	English-speaking parents of children <36 months referred to early intervention for developmental concerns	88% AA parents. Separate analysis of low health literacy group
Jibaja-Weiss et al 2011	Urban-Sth USA	138		✓				✓				BCT vs. mastectomy	Women diagnosed with early stage BCa (I-IIIA), candidates for breast-conserving surgery	38% AA, 45% Hisp., 100% uninsured
Kuppermann et al 2009	California USA	496		✓							✓	Prenatal genetic testing	Pregnant women, ≤20 weeks gestation who have not undergone prenatal testing	15.5% AA, 17.7% Lat., 13.3% Asian, 5.6% Other. Separate analysis to test moderating effects of educ. and site
Kuppermann et al 2014	California USA	710	✓	✓			✓					Prenatal genetic testing	Pregnant women, ≤20 weeks gestation who have not undergone prenatal testing	45% Lat., 16% AA, 59% from sites serving women of low-SES
Lepore et al 2012	Urban NE USA	490		✓	✓							PSA testing for PCa	Men aged 45–70, of AA descent, without PCa diagnosis, or testing <12 months prior	100% AA, 63% ≤ high school
Marteau et al 2010	England	1272									✓	Diabetes screening	Men and women aged 40–69 at risk of type 2 diabetes	Separate analysis to test moderating effects of SES
Miller et al 2011	Urban-Sth USA	264	✓	✓	✓	✓	✓				✓	CRC screening	Age 50–74, due for screening, SES disadvantaged	70% income <$20k, 73% AA, 67% ≤high school, 56% limited health literacy. Separate analysis to test moderating effects of health literacy
Miller et al 2018	Nth Carolina USA	450	✓				✓				✓	CRC screening	Age 50–74, English-speaking, scheduled to see primary care physician, due for screening	Separate analysis to test moderating effects of income, health literacy and race
Myers et al 2005	Urban-NE USA	242		✓	✓							PSA testing	Men, AA, 40–69, eligible for screening	100% AA, 61% ≤high school
Rising et al 2017	USA	898				✓					✓	Chest pain testing vs. observation and follow-up	Age 18+, presented to emergency with chest pain, negative cardiac workup, no ischemic ECG changes, cardiac troponin < upper limit of normal	> 61% low health literacy. Separate analysis to test moderating effects of race, income, insurance, education, health literacy and numeracy
Ruffin et al 2007	Midwest USA	174					✓				✓	CRC screening	Age 50–70, due for screening, no previous screening	Separate analysis to test moderating effects of geographical location, insurance, education and race
Schroy et al 2011	Urban NE USA	666		✓				✓				CRC screening	Age 50–75, due for CRC screening	63% AA, 6% Hisp., 66% Medicaid, Medicare, free care, or none
Smith et al 2010	NSW Australia	572	✓		✓							Bowel cancer screening	Age 55–64, English-speaking, average or slightly above average risk of bowel cancer, low education	59% 0–10 years in education
Street et al 1995	Urban-Sth USA	60									✓	BCT vs. mastectomy	Females with stage I or II BCa	Separate analysis to test moderating effects of education
Taylor et al 2006	USA	238		✓								PCa screening	Men aged 40–70, no history of prostate cancer	100% AA. Separate analysis to test moderating effects of education
Trevena et al 2008	Australia	314			✓						✓	CRC screening	Age 50–74, good English, no personal history of CRC	78% <high school. Separate analysis to test moderating effects of education
Vina et al 2016	USA	493	✓	✓								Knee replacement	Age 50+ years, AA, chronic knee pain, evidence of knee osteoarthritis	100% AA, 47% <$15k income
Volk et al 2008	Urban-Sth USA	450									✓	PSA testing	Male primary care patients, age 50–70 if not AA, 40–70 if AA, no history of PCa	Separate analysis to test moderating effects of low literacy
Williams et al 2013	Urban-Sth USA	543		✓							✓	PSA testing	Men aged 40–70, no PCa history, pre-registered for screening >5 days in advance, English-speakers	61% AA. Separate analysis to test moderating effects of race
Wolf et al 1996	Urban/Rural NE/Sth USA	205	✓		✓			✓				PSA testing	Men ≥50 years, no PSA testing or PCa history	65% income <$15k, 68% <high school, 59% public insurance

AA=African American, Lat.=Latina, Hisp.=Hispanic CRC=colorectal cancer, PSA=prostate-specific antigen, PCa=prostate cancer, BCa=breast cancer, BCT=breast conserving therapy, SES=socioeconomic status

Figure 1.

Inclusion and exclusion of studies in 2014–2020 versions of the Cochrane review^*

Most included studies evaluated unique PtDAs. Exceptions include two studies by Kuppermann et al.^21,22 that evaluated the same PtDA titled ‘Prenatal testing: exploring your options’. Three studies evaluated variations of a previously validated PtDA which included two videos^23,24 and a computerized program titled ‘CHOICE’ (Communicating Health Options through Interactive Computer Education).²⁵

Studies were conducted in the United States (n=22), Australia (n=2) and England (n=1; see Table 2). Participants included people considering prostate-specific antigen testing for prostate cancer screening (n=6), colorectal cancer screening (n=8), people with osteoarthritis considering knee replacement (n=2), women with early stage breast cancer considering surgery options (n=2), men with prostate cancer deciding about treatment (n=1), and people with chest pain deciding to be admitted for further testing or followed-up (n=1). Participants also included those considering living donor kidney transplantation (n=1), prenatal genetic testing (n=2), diabetes screening (n=1), or early intervention for children with developmental concerns (n=1).

Characteristics of disadvantaged populations relative to the reasons for inclusion are shown in Table 2. Studies included participants from disadvantaged groups including at least 50% of the sample with lower literacy (n=3) or lower education (n=7), or who were socially disadvantaged with respect to poverty or socioeconomic status (n=6), ethnicity or race (n=16), geographical location (n=2), or insurance status (n=5). Two studies recruited socioeconomically disadvantaged participants by targeting particular postcodes.^26,27 Three studies also implied that the majority of their sample included participants from disadvantaged groups as they recruited from disadvantaged geographical locations or sites that primarily serve disadvantaged groups.^24,25,28 Studies that conducted separate analyses explored the moderating effects of literacy or health literacy (n=5), education (n=8), numeracy (n=1), poverty or socioeconomic status (e.g. income, n=2), ethnicity or race (n=5), geographical location (n=2), insurance status (n=3) or language preference (n=1) by including interactions. One study that conducted a separate analysis only restricted their analysis within the low-literacy group, but did not compare to a high literacy group²⁹ and another conducted a separate analysis to test the effect of their intervention among Latino participants only.²³

Risk of bias

Risk of bias assessment results are presented in the Appendix, and in our companion paper (Yen et al., in this issue).

PtDA format and implementation

Formats of PtDAs varied but were most commonly computerized or web-based multimedia programs that included audio, video and interactive learning modules (n=10 studies; Table 3).^{21,22,24,25,28,30–34} Other formats included print (n=4),^26,35–37 video (n=3),^23,38,39 and audio (n=1).⁴⁰ Often interventions included a combination of formats such as print/video (n=3).^27,41,42 Four studies used print/video with another element such as text messages,²⁹ telephone,^43,44 or face-to-face counselling.⁴⁵ It was generally not specified whether a modality was chosen specifically to accommodate users with low literacy, with the exception of two studies that reported using a touch screen PtDA format to this end.^24,34

Table 3.

Methods used to tailor PtDAs for disadvantaged groups

	Methods of tailoring for disadvantaged groups
Author and year	Reading age calculated	Strategies to reduce cognitive demand	Use of media other than text	Consumer involvement in development	User testing with disadvantaged groups	Language adaptation	Communication/literacy experts	Decision-aid format and strategy used to tailor it for low health literacy/disadvantaged group	Brief description of comparator/control group
Boulware et al 2018^	✓		✓	✓	✓		✓	1) PREPARED: 50-min DVD describing living donor kidney transplant and a 162-page book written at 4th grade reading level summarizing the evidence about treatment effects on aspects of patients’ lives. Focus groups of patients, families, experts (clinicians, health education specialist, patient advocacy specialist, video scriptwriter, medical illustrator). Cognitive interviews with patients. Two pilot phases with 48 diverse patients. 2) PREPARED PLUS: offered donors reimbursement along with PtDA video and book.	Routine care in hemodialysis facilities. Participants in any 3 groups could have received educational materials or financial assistance through usual care
Brenner et al 2016^#		✓	✓	✓	✓	✓		Video: Communicating Health Options through Interactive Computer Education (CHOICE): 15 mins, CRC screening overview, comparison of test options and selection of a colored brochure to correspond to readiness. Spanish translation using original PtDA appraisal (Miller et al 2011), literature review, assessment of demographic trends, solicitation of information from target population. Trained navigators provided tailored support immediately after clinician encounter.	Attention control: food safety video before encounter and usual care after encounter
Diefenbach et al 2018^	✓		✓				✓	Healing Choices: multimedia software on CD-ROM including information about PCa treatment, ethnically diverse patient videos, physician’s view of treatment and recovery, opportunity to determine values/preferences. Theoretical frameworks and literature review guided content selection. Development included conceptual, literacy and cultural appropriateness review, user and usability testing, 7th grade reading level, guided by literacy and patient education experts.	Usual care: spoke to specialist who answered questions, received standard CIS and NCI print materials
Hoffman et al 2017		✓	✓					Video: 30mins, colorectal cancer screening educational and decision support, encouraging patients to talk to their PCP, ask questions and share concerns and preferences using AA family story. Content review by experts, prototyping, video production, pilot testing using cognitive interviews. Incorporated IPDAS, Ottawa Decision Support Framework, and Integrated Model of Behavior. EDAM used to improve saliency for AAs and ensure PtDA was accessible across literacy levels.	Attention control: hypertension video providing similar educational content but lacking decisional support and tailored components
Ibrahim et al 2017			✓					Video: 40mins, discusses knee osteoarthritis treatment options (lifestyle changes, medications, injections, complementary therapy and surgery, risks, benefits and known efficacy of each option, clinical indications, rehabilitative care, recovery time, effort and cost). Developed by Foundation for Informed Medical Decision Making. Same PtDA tested by Vina 2016.	Educational booklet developed by the National Institute of Arthritis and Musculoskeletal and Skin Diseases
Jibaja-Weiss et al 2011^		✓	✓		✓	✓		Interactive computerized web soap opera episodes walk women through BCa journey, episodes adapted to linguistically, race/ethnically and age targeted to user. EDAM utilized. Special design considerations for low literate computer users: voice-over navigational instructions, limited on-screen text. Developed with expert panel, usability interviews with target population, user testing in subgroups. Adapted episodes to Spanish. Updated version followed IPDAS guidelines.	BCa treatment educational materials generally provided to patients in these hospitals
Jimenez et al 2017		✓	✓	✓				Video (3mins) + text message explaining child development and Early Intervention (EI). Focus groups and interviews with parents & primary care professionals. Followed IPDAS criteria and accommodated low health literacy users using strategies such as plain language, visual reinforcement of key ideas, summaries of key information, and provided specific action steps. Feedback from parents, leaders from EI agency and other experts in child development. SMS reminder based on parent interviews to augment DA intervention.	Standard care: additional age-appropriate handout, publicly available from the CDC “Learn the Signs. Act Early” campaign
Kuppermann et al 2009*			✓	✓		✓	✓	Interactive computerized web-program emphasizing decision is personal and depends on values, explains screening vs. diagnostic tests, risks/benefits, describes Down Syndrome, gives tailored estimates. Focus groups and pilot testing among women and genetic counsellors. Spanish translation using forward and backward translation process with three bilingual team members. Kuppermann 2014 evaluated same PtDA.	Computerized version of booklet provided to pregnant women by California Expanded AFP Screening Program obstetricians
Kuppermann et al 2014			✓	✓		✓	✓	‘Prenatal Testing: Exploring Your Options’: interactive computer program (audio, video and text elements) narrated by bilingual actress. Educational module (general information + role of values and preferences), features of screening and tests. Input from clinicians, decision scientists, communication/literacy experts, focus groups of women, pilot testing with women and genetic counsellors, Spanish translation process. Participants told study would pay for any tests discussed for which they did not have insurance coverage. Kuppermann 2009 evaluated same PtDA.	No intervention or financial support provided
Lepore et al 2012^	✓		✓	✓	✓			Information booklet “Prostate Cancer: Your Life – You Decide” and telephone counselling session (education about risk, benefits and harms of screening, values clarification and shared decision-making). Feedback from expert consultants and men in target subgroups, cognitive interviews to ensure comprehension. Pilot testing conducted with low income undeserved men. Written at reduced readability level. SMOG grade level of 7 and Flesch grade level of 2.7	Telephone education about fruit and vegetable intake guidelines + education pamphlet
Marteau et al 2010^	✓		✓	✓				Letter informing participants they are at risk of developing Type 2 diabetes and inviting them to screen. Provides diabetes information: risk factors and complications, screening process, possible benefits and harms and their likelihood of occurring. Words, numbers and pie charts used to convey risks and benefits. Developed iteratively using think-aloud approaches. Flesch-Kincaid Grade level = 5.76, reading age 11 or above. Text refined using readability tools.	Standard letter stating participant is at increased risk of diabetes
Miller et al 2011^			✓	✓	✓			Interactive web-based program (CHOICE): includes video clips, graphics and animations. Presents prevalence information, CRC screening rationale, description of tests. Content based on previously validated video guided by subgroup interviews; navigation designed for ‘low-literacy’ subgroup. User testing with participants with wide range of computer experience and education level.	Computer presented program about prescription drug refills and safety
Miller et al 2018^	✓	✓	✓		✓		✓	Video (8mins) reviewing fecal testing and colonoscopy (based on CHOICE PtDA). Delivered via mobile app on iPad ‘mPATH-CRC.’ Participants can self-order test on app, PCP notified if participant self-ordered. Participant received follow-up messages with information about test ordered. Content and navigation designed for people with low health and computer literacy (simple interface, audio narration). Team included experts in app development, health literacy and CRC screening. Usability study conducted in disadvantaged population. All material written at 6th grade level or lower.	Video: 4.3 mins about diet and exercise produced by Centers for Disease Control
Myers et al 2005*^		✓		✓	✓		✓	PCa information booklet, risk factors, symptoms, pros/cons of screening, follow-up tests and treatment options. Telephone decision counselling session where major decision factors elicited and participant identifies top three factors, entered into pre-programmed calculator and preference score shared with participant. Content field-tested by literacy expert in face-to-face interviews with sub-group. Feedback resulted in simplification of text. Decision counselling protocol pilot tested in primary care practices in target community.	Information booklet addressing PCa, risk factors, symptoms, pros and cons of screening, follow-up tests and treatment options
Rising et al 2017		✓		✓	✓			Patients provided with 100-person risk pictograph to depict personalized risk for acute coronary syndrome within 45 days. Provides explicit management options. Each patients’ PtDA color coded to correspond to level of risk to reduce cognitive demand. Iterative development process whereby research team and designers field tested with clinicians and patients in emergency (25% ≤high school). Participatory action research design included revisions based on feedback from patients and caregivers and ensured IPDAS criteria was met.	Usual care: clinicians instructed to have discussions regarding test results and treatment as per usual practice
Ruffin et al 2007^			✓	✓	✓			Interactive web-based education tool: ‘Colorectal Web’. Used graphics, text, and videos to encourage patients to weigh preferences to make an informed choice. Content and usability testing guided by focus groups and interviews conducted with participants from geographical areas with high rates of advanced CRC and a low incidence of CRC screening.	Non-interactive CRC website sponsored by Cancer Research and Prevention Foundation
Schroy et al 2011			✓	✓				Two intervention groups: 1) received interactive computer program using audio-visual and touch-screen format including descriptions of CRC screening methods, audio and visual comparisons, summary of test features, decision-making module. 2) received program and personal risk assessment. Subgroup focus groups of racially/ethnically diverse screened and unscreened men and women, assess out-of-pocket cost concerns, cultural sensitivity issues, usability tests.	9 Ways to Stay Healthy and Prevent Disease discussing generic lifestyle changes other than screening for minimizing risk of preventable diseases
Smith et al 2010^	✓	✓		✓	✓		✓	Information booklet + video presenting tailored risk information about the outcomes of screening for CRC with FOBT. Informed by interviews with adults of varying education/literacy level, experts in adult literacy and linguists. Plain language, basic design, illustrations and simple techniques to reduce cognitive effort (reduce amount of text, lay language, glossary, simplified diagrams, active voice, contextual information before factual). Existing PtDA adapted to reduce the Flesch-Kincaid Grade level from 10 to 7. Pilot testing using same recruitment strategy.	Standard consumer information booklet Flesch-Kincaid readability score grade 9
Street et al 1995*			✓					Interactive multimedia program: Options for Treating Breast Cancer, including understanding the problem, treatment options (mastectomy and lumpectomy with radiation), and experiences of other women who had undergone either mastectomy or lumpectomy.	Care of Patients with Early Breast Cancer brochure (8 pages).
Taylor et al 2006^		✓	✓	✓			✓	Two intervention groups: 1) Booklet: 16-page, information about PCa screening, questions to ask doctor, glossary, celebrity spokesperson. 2) Video: 25-mins, depicts middle-aged AA man, discussing screening with friends, family, doctor; decision process (attempts to understand, fears, doubts, questions). Eight focus groups conducted with AA men and two with clinicians and researchers. Participants provided feedback. Research team included urologists, nurse, Public Health educators, AA men, and behavioral science staff.	Wait-list: single pre-intervention interview occurred 1 month following randomization but before receipt of intervention
Trevena et al 2008	✓							Information booklet (20pages): CRC screening definition, baseline risk information over next 10 years, mortality reduction from screening, outcomes from biennial FOBT, information about how to collect FOBT. Flesch-Kincaid readability score was grade 10. Incorporated expert and lay beliefs about core issues for informed choice about FOBT obtained from previous research, including studies conducted by the authors.	Consumer version of Aus guidelines mailed consisting of 3 pages of text with information about biennial FOBT. FK readability grade 9
Vina et al 2016^			✓					Two components: 1) Video (40mins) - see Ibrahim 2017. 2) Motivational interviewing: ~30mins session, face-to-face, motivational interviewing by trained, certified interventionists. Participants asked thoughts about total knee arthroplasty, goals/preferences regarding arthritis, and provided information and support for discussing with PCP. Same PtDA evaluated by Ibrahim 2017.	Educational booklet developed by NIH National Institute of Arthritis and Musculoskeletal and Skin diseases.
Volk et al 2008*		✓	✓		✓			Interactive DVD/CD-based soap opera episodes: 53–68mins, EDAM design, didactic soap-opera episodes integrated with interactive learning modules to complement content, navigational instructions provided, ethnicity of main character tailored to viewer, story took character through process of PCa screening decision. Content tailored using racial/ethnic concordance and social matching, acceptability tests conducted in subgroups.	Audio-booklet: same content, accompanied by narration. Lacked interactivity of intervention, testimonials and values clarification exercise
Williams et al 2013	✓			✓	✓		✓	Booklet (24 pages): Prostate Cancer Screening: Making an Informed Decision. Includes information on leading causes of death among men, accuracy of PSA test, PCS guidelines and diagnostic procedures and treatments. Adapted to 8th grade readability by plain language experts, usability tests in subgroups.	Usual care educational materials – National Cancer Institute’s 3-page fact sheet.
Wolf et al 1996					✓			Script about PSA screening read aloud by research assistant: overview of PSA screening, stated lifetime probability of developing and dying from PCa, known risk factors, ability of PSA test to detect early asymptomatic cancer, description of management options and major complications. Content developed by physician experts, piloted with primary care patients via subgroup interviews, assessed for comprehensibility.	Brief statement: blood test known as the PSA is available that can sometimes detect early prostate cancer before it is otherwise apparent

PCP=Primary Care Physician, IPDAS=International Patient Decision Aid Standards, ODSF=Ottawa Decision Support Framework, EDAM=Edutainment Decision Aid Model, AA=African American, PCa=prostate cancer, CRC=colorectal cancer, BCa=breast cancer, PSA=prostate-specific antigen, AA=African American, FK= Flesch-Kincaid

^*Included in 2014 Cochrane Review of PtDAs but excluded from 2017 update due to comparing simple versus detailed PtDAs

^{^}Development reported in separate paper

^#Decision-making outcomes reported in Brenner et al., 2016, implementation outcomes reported in Reuland et al., 2017

Tailoring for adults with low health literacy and other disadvantaged groups

Table 3 lists strategies for tailoring PtDAs to disadvantaged groups. In total, eight studies (33%) reported assessing readability to improve their PtDas, which represents the minimum requirement to meet IPDAS quality criteria. Seven studies (28%) reported designing their PtDA at a reading level of grade ≤8, four of which reported involvement of experts in either plain language, literacy, adult education or health communication.^27,30,36,42 Readability analysis software was used in one study to refine the PtDA text,²⁶ one study reported the text was written at reduced readability,⁴³ and one study simply reported that the reading grade level was calculated, however it is not clear whether it was specifically designed to be below grade 8.²⁴ Four studies stated that IPDAS criteria were followed; three attended to the original IPDAS quality criteria checklist,^29,37,42 while one attended to the IPDAS instrument (IPDASi v3.0) checklist.²⁷ An additional study reported that the IPDAS guidelines contributed to the conceptual framework that guided PtDA development,³⁸ and another updated their PtDA after the study was conducted to satisfy the original IPDAS quality criteria checklist.³¹ Eight studies reported using strategies to reduce cognitive burden including plain language, the use of a glossary of key terms, bullet points, visual cues and illustrations, narration, and simple navigation.^{23,27,29,31,32,37,38,41,44} Three studies used the Edutainment Decision Aid Model (EDAM) to guide development of PtDAs.^31,32,38 This approach was initially developed in 2007 by Jibaja-Weiss and Volk with the aim to make a PtDA both entertaining and educational for users with low-literacy by incorporating tailored soap opera scenes and linking them to interactive learning modules.⁴⁶ In the PtDAs that utilized EDAM, the characters in the soap opera episodes were tailored to the ethnicity/race and age of the user.

Studies that used other methods of tailoring for disadvantaged groups are specified in Table 3. Of note, sixteen studies reported involvement of consumers, whereby two reported having a patient partner on the research team^41,42 and fifteen involved patients/consumers in the development of the PtDA. Seven reported conducting focus groups with patients,^{21–23,34,36,41,42} three reported conducting interviews with patients,^25,27,44 and two reported conducting both focus groups and interviews.^28,29 Thirteen studies reported conducting pilot, user or usability testing with target disadvantaged groups (Table 3). Five additional studies conducted a form of user testing but did not specify whether it was in disadvantaged groups.^{21,22,30,34,38} Communication or literacy expert involvement was reported in nine studies, indicated in Table 3.

Cognitive demand: readability, comprehension and actionability scores of PtDAs

We were able to access 11 PtDAs out of the total 24 unique PtDAs from journal articles or supplementary material (n=2), decision-aid repositories (n=2) or via direct contact with the PtDA author (n=7). For two of these, we were only able to access a component of the intervention; an information booklet evaluated by Myers et al. (accompanied by motivational interviewing),⁴⁴ and the video component of an iPad program evaluated by Miller et al.²⁴ Of those obtained, the readability, understandability and actionability is shown in Table 4. Using our measure of readability, none of the PtDAs were written at or below 8^th grade reading level.

Table 4.

Readability and Patient Education Materials Assessment Tools (PEMAT) scores for PtDAs which could be accessed

Authors and year	Readability score (grade level)*	PEMAT score# Score >70% = suitable for lower literacy popln
		Understandability (%)	Actionability (%)
Print
Boulware et al 2018	9.07	94.1	60.0
Marteau et al 2010	8.98	81.3	66.7
Myers et al 2005	9.99	87.5	66.7
Rising et al 2017	12.66	75.0	75.0
Smith et al 2010ǂ	8.47	93.8	83.3
Taylor et al 2006^	11.49	75.0	50.0
Trevena et al 2008	11.54	87.5	60.0
Audio-visual (computerized program or video)
Boulware et al 2018	12.82	83.3	100.0
Brenner et al 2016	11.23	84.6	75.0
Jibaja Weiss et al 2011+	10.60	92.3	100.0
Miller et al 2018	11.49	84.6	50.0
Taylor et al 2006^	8.25	60.0	33.3

^*readability scores are the average of the SMOG and Gunning Fog index. High scores indicate the PtDA is harder to read.

^#print PtDAs assessed using the PEMAT-P (printable materials), video and computerized PtDAs assessed using the PEMAT-A/V (audiovisual materials)

^ǂWe did not assess the video component of the PtDA evaluated by Smith et al., as we were informed by authors that the content was identical

⁺PtDA updated after the study to meet IPDAS quality criteria checklist, version used for readability and PEMAT analysis may differ from the version evaluated in the study

^{^}Both print and audio-visual materials were assessed for Taylor et al 2006

Wolf et al 1996 excluded from readability and PEMAT analysis due to format of PtDA

Using the PEMAT measure, understandability scores were high with 92% (n=10) achieving the recommended threshold (Table 4). However, only 42% (n=5) of PtDAs achieved the recommended threshold for actionability. The mean understandability score was 83.3% (SD=9.3%) and the mean actionability score was 68.6% (SD=20.2%).

Attitudes towards and acceptability of PtDAs

Volk et al. tested the moderating effects of literacy level on the acceptability of two prostate cancer screening PtDAs.³² Compared to those with high literacy, low literacy participants were more likely to rate the entertainment aid as having too much information, but also less likely to rate it as too long and were more likely to ask questions overall. Low literacy participants rated the clarity of the audio-booklet higher than the entertainment aid but were more engaged with elements of the entertainment aid. Kuppermann et al. tested the moderating effects of education on intervention satisfaction. They found that all women were more likely to say they would recommend the PtDA to a friend, but the increase was more pronounced among women with a bachelor’s or graduate degree.²¹

Subgroup analyses of the impact of attention to health literacy in PtDA design on decision making outcomes (knowledge and decisional conflict)

The meta-analysis examining the impact of PtDAs on decision-making and health outcomes is reported in the companion paper in this issue (Yen et al., in this issue). We conducted a subgroup analysis of knowledge (scale of 0–100) according to whether readability was assessed and reported (Figure 2) and found no significant differences in the impact of PtDAs on knowledge between studies that reported a reading age (pooled mean difference (MD)=9.80 (95% CI 1.92, 17.68); I²=96%, p=0.01) and studies that did not (pooled MD=15.62 (95% CI 9.24, 22.00); I²=93%, p<0.000), (Chi²=1.27, p=0.26; I²=21%). For decisional conflict, there was also no difference in the impact of PtDAs between studies that reported reading age (pooled MD=−3.24 (95%CI −7.54, 1.07); I²=57%, p=0.14) and studies that did not (pooled MD=−5.38 (95%CI −10.16, −0.60); I²=87%, p=0.03), (Chi²=0.43, p=0.51; I²=0%; Figure 3). The subgroup analysis of knowledge according to whether strategies to reduce cognitive demand were reported (Figure 4) indicated PtDAs led to greater knowledge improvements in studies that did report using strategies to reduce cognitive demand (pooled MD=20.96 (95% CI 16.06, 25.85); I²=77%, p<0.000) compared to studies that did not (pooled MD=8.65 (95% CI 4.50, 12.81); I²=91%, p<0.000); (Chi²=14.11, p=0.0002, I²=92.9%). For decisional conflict, there was no significant difference in the impact of PtDAs between studies that reported using strategies to reduce cognitive demand (pooled MD=−11.17 (95% CI −26.25, 3.91); I²=91%, p=0.15) and studies that did not (pooled MD=−1.96 (95%CI −3.67, −0.25); I²=43%, p=0.02); (Chi²=1.42, p=0.23, I²=29.4%; Figure 5).

Figure 2.

Subgroup forest plot for knowledge according to the reporting of reading age.

Figure 3.

Subgroup forest plot for decisional conflict according to the reporting of reading age.

Figure 4.

Subgroup forest plot for knowledge according to use of strategies to reduce cognitive demand.

Figure 5.

Subgroup forest plot for decisional conflict according to use of strategies to reduce cognitive demand.

DISCUSSION

As part of the IPDAS 2.0 process, we sought to update a review of health literacy and PtDAs published in 2013.⁷ Specifically, we examined the extent to which PtDAs are designed to meet the needs of low health literacy and disadvantaged populations. This complements an additional review which examined the impact of these PtDAs on decision-making and health outcomes (Yen et al., in this issue). The included studies comprised almost 10,000 patients/community members and indicated that, of the 213 PtDAs developed and included as part of the Cochrane review and updates since 2006, only 11% of PtDAs (n=24 tools published in 25 papers) were developed for, or evaluated with, patients and populations who are disadvantaged with respect to health literacy, education, income, race/ethnicity, neighborhood or health insurance. This represents an increase from 5% reported in 2013 (although note that slightly different criteria were used for this search),⁷ yet is still remarkably low given the number of patients in the health system who are disadvantaged and likely to require health decision support.

Of the PtDAs that were developed and evaluated, only one third (8/25) adhered to the IPDAS instrument and IPDAS minimum standards by reporting any readability statistic. Only 7 PtDAs reported that they met the Grade 8 reading level criteria (the reading level recommended for an average population sample), a requirement of the original IPDAS quality criteria checklist. However, among the subset that were available to be assessed independently, none met the criteria when independently reviewed. This is partly due to the methods we used in our independent review process for preparing and analyzing the PtDA text. In particular, the readability calculator we used is considered superior to other readability calculators (as recommended by health literacy experts) but has not been widely used by PtDA developers. However, our findings in this regard speak to the need for consistency across studies in the use of readability instruments and/or greater transparency in reporting of methods of calculation.

In our assessment of understandability and actionability using the PEMAT, which requires multiple raters to make a more subjective assessment of materials, we found that the PtDAs scored well on the understandability domain but poorly on the actionability dimension of the scale. The poor rating on actionability of health materials is a common finding as they often fail to pay attention to the clarity of instructions and actions for people/patients to follow.^11,47 This may reflect the fact that many PtDAs focus on the decision itself, not the actions or steps that follow the decision. The observed discrepancy between more simple but objective measures of readability versus more sophisticated but subjective ratings of comprehension are not unexpected. However, efforts to overcome subjectivity in PEMAT are addressed by the use of two or more raters. This measure also correlates with readability scores and patient reported comprehension and actionability rating and has the advantage of including an audiovisual dimension.¹⁹ In total our results suggest there is room for significant improvement in the design and reporting of decision aids for lower literacy populations.

In addition to the assessment of readability, developers of PtDAs included in our review used a range of strategies to tailor their PtDAs to disadvantaged patients and populations. The main strategies included working with patients or consumers in various ways (e.g., pilot or user testing and focus groups or interviews), the involvement of adult education, communication, or literacy experts, and the use of PtDA formats other than text (e.g., video or multimedia programs). However, few studies reported including patients as partners. It is important as research goes forward that patients are included as equal partners in research and that pilot testing (and other forms of tailoring / gathering feedback) is conducted and clearly reported with patients/consumers from the target disadvantaged groups. This was often not made clear in the included studies.

Building on a concurrent meta-analysis and narrative synthesis exploring the impact of PtDAs on decision-making and health outcomes (Yen et al., in this issue), we were able to conduct subgroup analyses to examine whether the impact of decision quality outcomes - knowledge and decisional conflict - differed according to the PtDA tailoring strategies reported. Our findings highlighted that PtDAs may result in greater knowledge improvements when tailoring strategies to reduce cognitive demand (e.g., plain language, bullet points, visual cues and illustrations) are used in developing the PtDA. We did not find any differences in the impact of PtDAs on knowledge between studies that reported calculating reading age and studies that did not, possibly due to differences in methods used to assess readability. There were also no significant differences in the impact of PtDAs on decisional conflict between studies that reported tailoring strategies and those that did not, however, only three studies could be pooled in each subgroup.

Strengths and limitations of this review

We built on and strengthened our previous review by including 25 RCTs and including a systematic review of the attention paid to health literate design. We were also able to strengthen our previous review by conducting subgroup meta-analyses to determine any differing impact of PtDAs on knowledge and decisional conflict as a result of using strategies to tailor PtDAs for low health literate populations. We used the new version of the Cochrane Risk of Bias tool and conducted screening and risk of bias assessments with two independent reviewers. Data was extracted independently by two reviewers.

To assess readability of the PtDAs, we followed guidelines from Health Literacy Connections.⁴⁸ This included preparing the text as detailed in the methods and calculating the estimated reading grade level using the average of the Gunning Fog index and the SMOG index. Several included studies reported designing the PtDA at reduced reading grade level and calculating readability statistics of PtDAs in the development process. For many of these PtDAs, there are notable discrepancies between the readability statistics reported in the studies and those we calculated in table 4. This could be explained by factors including variations in the preparation of the text (e.g., removing headings, periods that do not indicate end of sentences, sentence fragments, bullet points not in full text), readability formula used (e.g., Flesch Kinkaid Grade Level has been reported to produce reading grade levels 2–3 lower),⁴⁹ and/or method of conducting the analysis (e.g., using an online tool, Microsoft word function or calculating manually using the formula). Due to the subjective nature of the PEMAT, the PtDAs were dual coded by JS and OM, with any discrepancies resolved with discussion with KM. The actionability domain of the PEMAT is limited in this context, as there may not always be an explicit action to take. Furthermore, for two studies we were only able to access a component of the intervention, which may limit the accuracy of the actionability scores.

We deliberately adopted broad eligibility criteria to include studies with ‘disadvantaged populations’, not only lower health literacy on account of the fact that health literacy is known to correlate with socially-defined factors (such as educational attainment; socioeconomic status) in most societies across the world.^3,14 However, we did not consider the studies that included older adults (that did not report any other measures of disadvantage) despite the fact that this population may comprise a greater proportion of people with low health literacy skills compared to the general population. Therefore, we acknowledge that some studies – and associated strategies to simplify PtDAs – may have been missed.

Recommendations to IPDAS

On the basis of our review, we make the following recommendations for consideration in future IPDAS criteria related to health literacy:

• All PtDAs report readability scores and in addition use and adhere to PEMAT thresholds for comprehension and actionability of written and audiovisual documents.

• Readability scores are assessed using utility.com and suggest where possible PtDA developers develop tools at reading grade level 5/6 consistent with a Universal Precautions approach and the AHRQ Health Literacy Universal Precautions Toolkit.⁵⁰

• Patient partners are included in all studies with low literacy populations

• Studies that target disadvantaged populations must report how PtDAs have been piloted with their target group to ensure needs and preferences are addressed.

• In line with good principles of health communication we recommend the use of audiovisual, audio as well as written health information and the provision of translated materials. We also recommend using specific strategies to reduce cognitive demand such as basic design, lay language, glossaries, simple narration, visual reinforcement of key ideas, summaries, limited on screen text and simplification of text, presentation of contextual information before factual, and the use of active voice.

Directions for future research

By conducting this review, we were able to gain a preliminary understanding of how the use of tailoring strategies may improve knowledge and decisional conflict. However, there were not enough studies to make meaningful comparisons for other outcomes or other tailoring strategies, and conclusions from these subgroup analyses are limited due to the lack of details provided in some studies, as well as the inconsistency in defining such strategies (e.g., consumer involvement, user testing). Our hope is that the quality of reporting of the development and evaluation of PtDAs may improve in future with the use of the SUNDAE checklist,⁵¹ as there is a need for greater consistency in reporting of development strategies and the measurement of outcomes in order to conduct more meaningful analyses. In future, we may be able to better clarify what tailoring strategies are used and their impact on decision-making and health outcomes. Finally, little research has addressed how disadvantaged populations best access PtDAs. This is vitally important if we truly wish to improve equity with respect to accessing decision support.

Box 2. Overview of Patient Education Materials Assessment Tool (PEMAT; Print and AV Versions).

Domain: Understandability (17 items P; 13 items AV)

Topic: Content

Item 1: Makes its purpose completely evident (P and A/V)

Item 2: Does not include information or content that distracts from its purpose (P)

Topic: Word Choice and Style

Item 3: The material uses common everyday language (P and A/V)

Item 4: Medical terms are defined and used only to familiarise readers (P and A/V)

Item 5: The material uses the active voice (P and A/V)

Topic: Use of Numbers

Item 6: Numbers are clear and easy to understand (P)

Item 7: The material does not expect the user to perform calculations (P)

Topic: Organization

Item 8: The material breaks or ‘chunks’ information into short sections (P and A/V)

Item 9: The material’s sections have informative headers (P and A/V)

Item 10: The material presents information in a logical sequence (P and A/V)

Item 11: The material provides a summary (P and A/V)

Topic: Layout and Design

Item 12: The material provides visual cues to draw attention to key points (P and A/V)

Item 13: Text on the screen is easy to read (A/V)

Item 14: The material allows the user to hear the words clearly (A/V)

Topic: Use of Visual Aids

Item 16: The material uses visual aids whenever they could make content more easily understood (P)

Item 17: The material’s visual aids have clear titles or captions (P)

Item 18: The material uses illustrations and photographs that are clear and uncluttered (P and A/V)

Item 19: The material uses simple tables with short and clear row and column headings (P and A/V)

Domain: Actionability (7 items P; 4 items AV)

Item 20: The material clearly identifies at least one action the user can take (P and A/V)

Item 21: The material addresses the user directly when describing actions (P and A/V)

Item 22: The material breaks down any action into manageable, explicit steps (P and A/V)

Item 23: The material provides a tangible tool whenever it could help the user take action (P)

Item 24: The material provides simple instructions of how to perform calculations (P)

Item 25: The material explains how to use charts, graphs, tables or diagrams to take actions (P and A/V)

Item 26: The material uses visual aids whenever they could make it easier to act on instructions (P)

A/V, audiovisual; P, print.

APPENDIX

Table 1.

PEMAT scoring

Item #	PEMAT items	‘Agree’ responses n (%)
	Understandability
	Content
1	Makes its purpose completely evident (P and A/V)	12 (100)
2	Does not include information or content that distracts from its purpose (P)	7 (100)
	Word Choice and style
3	Common everyday language (P and A/V)	12 (100)
4	Medical terms are defined and used only to familiarise readers (P and A/V)	12 (100)
5	Active voice (P and A/V)	12 (100)
	Use of numbers
6	Numbers are clear and easy to understand (P)	7 (100)
7	Does not expect the user to perform calculations (P)	7 (100)
	Organization
8	Chunks information into short sections (P and A/V)	10 (83)
9	Sections have informative headings (P and A/V)	8 (67)
10	Presents information in a logical sequence (P and A/V)	12 (100)
11	Provides a summary (P and A/V)
	Layout and design
12	Provides visual cues to draw attention to key points (P and A/V)	10 (83)
13	Text on the screen is easy to read (A/V)*	4 (100)
14	The material allows the user to hear the words clearly (A/V)	5 (100)
	Use of visual aids
15	Uses visual aids whenever possible (P)	3 (43)
16	Visual aids reinforce rather than distract (P)	7 (100)
17	Visual aids have clear titles and captions (P)	6 (86)
18	The material uses illustrations and photographs that are clear and uncluttered (P and A/V)*	11 (100)
19	Tables are simple with short, clear row and column headings (P and A/V)^	4 (100)
	Actionability
20	Clearly identifies at least one action for the user to take (P and A/V)	12 (100)
21	Addresses the user directly when describing actions (P and A/V)	11 (92)
22	Breaks down actions into manageable, explicit steps (P and A/V)	8 (67)
23	Provides a tangible tool whenever it could help the user take action (P and A/V)	7 (100)
24	Provides simple instructions or examples of how to perform calculations (P)#	-
25	Explains how to use the charts, diagrams, graphs, or tables to take actions (P and A/V)~	3 (33)
26	Uses visual aids whenever possible to help take action (P)	1 (14)

^*N/A for 1 PtDA

^{^}N/A for 8 PtDAs

^#N/A for all PtDAs

^~N/A for 3 PtDAs

Risk of bias assessment

Of the nine studies we assessed for risk of bias, four had unclear risk of bias overall (Table 2). This was mainly due to concerns regarding deviations from the intended interventions (n=8), outcome measurement (n=4) and reporting results (n=5). These concerns typically arose because there was no blinding of participants or researchers delivering interventions, lack of information about blinding of outcome assessors or no evidence of planned analyses. Of the sixteen studies that were already assessed as part of the 2014 and 2017 versions of the Cochrane review (Table 3), three had low risk of bias for all domains, but many had unclear risk of bias for allocation concealment (n=11), blinding of participants (n=9) and selective reporting (n=10).

Table 2.

Risk of bias summary for each included study not included in 2014/17 Cochrane review

Author and year	Randomization	Deviations from the intended interventions	Missing outcome data	Outcome measurement	Reporting results	Overall risk of bias
Boulware et al 2018	+	?	?	+	?	?
Brenner et al 2016	+	-	+	?	+	?
Diefenbach et al 2018	+	?	?	?	?	?
Hoffman et al 2017	+	?	+	?	+	+
Ibrahim et al 2017	+	?	+	+	+	+
Jimenez et al 2017	?	?	+	?	?	?
Miller et al 2018	+	?	+	+	?	+
Rising et al 2017	+	?	+	+	+	+
Vina et al 2016	+	+	+	+	?	+

⁺low risk of bias

^?some concerns

^-high risk of bias

Table 3.

Risk of bias summary for each included study in 2014 and 2017 Cochrane review

Author and year	Random sequence generation (selection bias)	Allocation concealment (selection bias)	Blinding of participants and personnel (performance bias)	Blinding of outcome assessment (detection bias)	Incomplete outcome data (attrition bias)	Selective reporting (reporting bias)	Other bias
Jibaja-Weiss et al 2011*	+	?	?	+	?	?	+
Kuppermann et al 2009^	?	+	+	+	?	?	+
Kuppermann et al 2014*	+	+	+	+	+	+	+
Lepore et al 2012*	+	?	?	+	+	+	+
Marteau et al 2010*	+	+	+	+	+	+	+
Miller et al 2011*	+	?	+	+	+	+	?
Myers et al 2005^	?	?	+	?	?	?	+
Ruffin et al 2007*	+	?	+	+	+	?	+
Schroy et al 2011*	?	?	?	+	+	?	+
Smith et al 2010*	+	+	+	+	+	+	+
Street et al 1995^	?	?	?	?	+	?	+
Taylor et al 2006*	?	?	?	?	+	?	?
Trevena et al 2008*	+	+	?	+	?	+	+
Volk et al 2008^	+	?	?	?	?	?	+
Williams et al 2013*	?	?	?	?	+	?	+
Wolf et al 1996*	?	?	?	+	+	?	+

⁺low risk of bias

^?unclear risk of bias

^-high risk of bias

^*Risk of bias assessed as part of 2017 version of Cochrane Review⁶

^{^}Risk of bias assessed as part of 2014 version of Cochrane Review¹³