Strategies to Optimize Comprehension of Numerical Medication Instructions: A Systematic Review and Concept Map

Abstract

Objective:

To develop evidence-based recommendations for improving comprehension of quantitative medication instructions.

Methods:

This review included a literature search from inception to November 2021. Studies were included for the following: 1) original research; 2) compared multiple formats for presenting quantitative medication information on dose, frequency, and/or time; 3) included patients/lay-people; 4) assessed comprehension-related outcomes quantitatively. To classify the studies, we developed a concept map. We weighed 3 factors (risk of bias in individual studies, consistency of findings among studies, and homogeneity of the interventions tested) to generate 3 levels of recommendations.

Results:

Twenty-one studies were included. Level 1 recommendations are: 1) use visualizations of medication doses for liquid medications, and 2) express instructions in time-periods rather than times per day. Level 2 recommendations include: validate icons, use panels or tables with explanatory text, use visualizations for non-English speaking populations and for those with low health literacy and limited English proficiency.

Conclusions:

Visualized liquid medication doses and time period-based administration instructions improve comprehension of numerical medication instructions. Use of visualizations for those with limited health literacy and English proficiency could result in improved outcomes.

Practice Implications:

Practitioners should use visualizations for liquid medication instructions and time period-based instructions to improve outcomes.

1. INTRODUCTION

Errors in medication self-administration have been documented in 1 out of every 10 emergency department visits for adverse drug events (1). The Institute of Medicine has concluded that one of the main causes of medication errors is poor patient comprehension of instructions, leading to the unintentional misuse of prescription medications (2, 3).

Medication instructions are challenging to interpret because they combine unfamiliar and complex pharmaceutical names with instruction information containing numerical information about dose, frequency and time, and other quantitative information such as maximum safe doses. As a result, they pose particular challenges for a general population, especially individuals with low health literacy, numeracy, or English proficiency (4, 5). A number of studies have demonstrated that these populations have higher rates of misinterpretation of instructions, which can lead to adverse medication events (6–10). In addition, children are particularly vulnerable to medication errors because of factors such as caregiver unfamiliarity of liquid medications and the complexity of weight-based dosing. One United States study found that between 2002–2012, an average of 63,358 children younger than six years of age experienced a medication error made out of the hospital annually. This breaks down to approximately once every 8 minutes (11).

Fortunately, ongoing research aims to improve medication instructions by incorporating illustrations or improving phrasing or formatting (2, 12) The studies in this review have tested different approaches of improving quantitative medication instructions. For example, Davis tested different ways of wording information about number of pills per day (13), and Yin tested pictograms (14). At a 2007 conference, the Institute of Medicine led the United States initiative on developing safer medication labels (15).These efforts, along with other international efforts, have resulted in the development of guidelines for optimizing medication instruction comprehension (15–26).

In this review we drill deep into an aspect of medication instruction that is particularly problematic – the best format for quantitative medication information. Current guidelines tend to be based on expert consensus that has incorporated available evidence. It is important to support these guidelines with continued research review. This paper will identify updated evidence-based findings through a systematic review to support or modify established guidelines, identify additional strategies to improve comprehension and highlight areas in need of further research (27).

2. METHODS

This study analyzes a subset of articles from a large systematic review of experimental and quasi-experimental research on mechanisms to present quantitative information in health. Our research team followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (28, 29) (Figure 1).

Figure 1. PRISMA Flow Diagram.

*Other reasons for exclusion include: No full text available, outcome indeterminable, education methods, outcome not of interest, decision was not a personal health/medical decision, non-patient (health professional).

2.1. Data sources and search

We performed the systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (30). In adherence to these guidelines, we registered a protocol in PROSPERO, an international prospective register of systematic reviews (registration #CRD42018086270). Two experienced librarians constructed a systematic approach to search Ovid MEDLINE, Ovid Embase, the Cochrane Library (Wiley), CINAHL (EBSCO), ERIC (ProQuest), PsycINFO (EBSCO), and the ACM Digital Library, from inception to January 2019, with an update on November 1, 2021. See Appendix A for the search strategy for Ovid MEDLINE. To supplement these results, we identified the top 4 most common journals from database searches (Medical Decision Making, Patient Education and Counseling, Risk Analysis, and Journal of Health Communication) and hand-searched their tables of contents in their entirety from 2008 up to 2021. For articles selected for inclusion in this study, we pulled and screened reference lists and citing articles from Scopus (Elsevier). Searches and de-duplication identified 27,628 articles that were screened by a team of researchers using Covidence systematic review web software (Covidence.org, Melbourne, Australia). We then assessed 1,555 articles for full-text review, with discrepancies resolved by consensus.

2.2. Eligibility criteria and selection

The initial search included numerical format studies that were head-to-head comparisons of different ways of presenting numerical concepts to patients or the public to improve communication of quantitative information, including “lab results, medical risk information, genomic data, drug labels and medication instructions”(27, 31). Studies were included in the current review if they met the following criteria: 1) original research; 2) compared multiple formats for presenting quantitative information in medication instructions; 3) included patients or lay-people as participants; 4) assessed comprehension-related outcomes quantitatively. Comprehension-related outcomes included objectively assessed understanding, demonstrated accuracy in medication administration, recall of instructions, adherence, or preference for one or more information formats. Each study was independently evaluated for eligibility by at least two members of a five-member study team. Disagreements were mutually discussed before final consensus. Our search terms identified papers within this larger group that are relevant to medication instruction and produced a subset of twenty-one papers.

2.3. Data extraction and data analysis

Study characteristics were extracted, including author, publication year, medication instruction information, intervention, outcomes, method used to assess outcomes, population, covariates of interest (literacy, health literacy, and English proficiency), associations between outcomes and covariates, and limitations. One team member conducted data extraction, which was reviewed and confirmed by a second team member. Team members met to resolve any conflicts.

2.4. Quality assessment

We adapted criteria established in the AHRQ Methods Guide for Comparative Effectiveness Reviews and Cochrane Handbook for Systematic Reviews of Interventions for assessing the risk of bias (32, 33). Criteria for bias concerns included characteristics of the sample selection (for example high level of education without stratification), randomization process, study protocols (for example, protocol deviations), measurement of covariates, missing data, and presence of other biases. Each study was categorized with an overall risk of bias based on identified levels of concern (Tables 8 and 9, Appendix A).

2.5. Development of evidence-based recommendations

The recommendations were developed based on three factors: risk of bias for the studies in a group, consistency among findings of the studies in that group, and homogeneity of the interventions examined in those studies. The first two factors (risk of bias and consistency) were used to determine strength of evidence (Table 1). Different measurement approaches precluded us from comparing point estimates and confidence intervals. An adaptation of the Grading of Recommendations Assessment, Development and Evaluation criteria for systematic reviews helped us develop three levels of strength.

Table 1.

Strength of evidence criteria

Strength of evidence within a group of studies
	Consistency of findings	Risk of bias within the group
Strong	High (All outcomes in same direction)	Little or none
Moderate	High	At least one with moderate, high, or unclear
	Moderate (Two or more, but not all, outcomes in same direction)	Little or none
Weak	Moderate	At least one with moderate, high, or unclear
	Low (Outcomes in different directions or findings from one study only)

The third factor was homogeneity of the interventions. In situations where studies had similar visualizations, we would class those as homogeneous. In situations where studies had thematically similar visualizations that differed in some respects, we would classify those as having lower homogeneity (Table 2). These determinations were made independently by two researchers and conflicts were resolved through mutual consensus.

Table 2.

Level of recommendation criteria

Level of evidence-based recommendation
	Strength of evidence	Homogeneity of the interventions
I (Intervention strongly supported)	Strong	High
II (Promising intervention, but more research needed)	Strong	Low
	Moderate	High
III (More research needed)	Moderate	Low or one study
	Weak	High, low, or one study

2.6. Categorization of the numerical medication interventions

To categorize the interventions in the included studies, we parsed them into quantitative concepts, communication elements and organizing structures (Table 3).

Table 3.

Concepts, elements, and structures

Quantitative Concepts	Communication Elements	Organizing Structures
• Frequency	• Words	• Table
• Time	• Numerals	• Timeline
• Dose	• Icons	• Panel

Quantitative concepts are the types of numerical information being communicated, including frequency, time, and dose. Frequency refers to the number of times to take a medication within a designated time interval. Time is a general concept that refers to intervals between doses, the time of day to take doses, or the duration of medication usage, and dose refers to the amount of medication taken.

Communication elements are features used to convey information, including words, numerals, and icons. Icons refer to small illustrations or symbols representing familiar objects, concepts, or functions. Rather than being extremely realistic (like a photograph), they are more figurative, and readers learn their meaning by applying pre-existing knowledge (34, 35).

Organizing structures are ways of presenting elements to communicate concepts. While these concepts could be presented in a sentence or paragraph, they could also be presented within one of 4 different organizing structures, such as a table, which is a structure arranged in rows and columns; a timeline, which is a graphical representation of a chronological sequence of events; or a panel, which is a set of rectangular fields that are not labeled as rows or columns and which may be organized to convey an overarching principle, such as the progress of time.

2.7. Concept maps

We developed a concept map to parse out the quantitative concepts and communication elements in each included study. Concept mapping has been shown to be an effective tool in identifying concepts, visualizing dependencies, and creating coherence in the writing literature reviews (36–39) (Figure 2).

Figure 2. Concept map for quantitative concepts and communication elements used in numerical medication instructions.

Note: * Icons were categorized as concrete (depicts a specific object fairly accurately) or abstract (depicts a concept more figuratively).

† Abstract icons were further categorized as direct (conveys a concept through an image of an object in general), implied (conveys a concept through an associated image), or arbitrary (conveys a concept through an image that has an association that must be learned or explained) (40, 41).

2.8. Communication interventions

We defined two major types of interventions used in the included papers: (1) format interventions, which compare variations in words and/or numerals only and (2) visualization interventions, which compare variations in different organizing structures (with or without icons).

1. Format interventions explore mechanisms of expressing quantities through numerals and non-numeric words, without images. For example, the timing of medications can be expressed as times per day (twice a day), frequency (every 12 hours), time periods (morning and bedtime), specific times (8 am and 8 pm), or mealtime anchors (with breakfast and dinner)(42).

2. Visualization interventions express quantitative information with or without icons. Other authors have referred to visual images used in medication instructions using different terms, such as “pictograms,” “icons,” “graphics,” or “pictorials”. For this paper, we refer to them as visualization interventions. We identified three major types of visualization interventions.

   1. Liquid dosing visualizations convey dosing information for liquid medications through icons plus words and/or numeric labels. They do not typically visualize time information as well (Figure 3).
   2. Maximum dosing visualizations use warning icons to draw attention to a maximum safe dose over a period of time (Figure 4).
   3. Time interval visualizations focus primarily on information about frequency and time, particularly when this is the most cognitively complex component of the instruction. These visualizations use organizing structures that frequently leverage natural mappings of left-to-right or up-to-down to express the progress of time from past to future (Figure 5).

Figure 3.

Liquid dosing visualization example

Figure 4.

Maximum dosing visualization

Figure 5.

Organizing structures used in time interval visualizations

3. RESULTS

3.1. Included studies and measured outcomes

Twenty-one studies met the inclusion criteria (Table 5). These studies covered prescription, over-the-counter, and hypothetical medications. A subset focused on liquid medication for children.

Table 5.

Included papers, in chronological order

Included studies	Sample Demographics Location (n=sample size)	Outcomes measured	Type of intervention	Communication elements	Comparator	Overall significant findings	Covariate significant findings	Limitations Risk of bias
Morrell et al. (60) 1990	Local church/Senior citizens groups College students US (n=64)	Comprehension Recall	Time interval visualization	• Panels • Direct icons of pills	Text alone (“Take 1 capsule 3 times a day”)	•Lowered comprehension for both young and old age groups. •Improved recall for younger group.	•Younger adults made fewer errors with pictogram for recall. •Pictograms appeared to hamper older adults’ memory for information. •Older adults recalled less information across both presentation conditions.	• Use of hypothetical medications •Study did not explore text plus icons Little risk of bias: •Unclear randomization mode •Unclear education level of participants
Morrow et al. (61) 1996	Senior citizens College students US (n=28)	Comprehension Recall Preference Time	Time interval visualization	•Timeline – 3 types 1. Horizontal timeline 2. 24-hour clock 3. 12-hour clock •Implied icons of sun/moon •Direct icons of pills	Text alone (Text not included)	•Visualizations lowered comprehension and had no influence on recall. •Order of the most preferred, and fastest: text-only, horizontal timeline, 24-hr, 12-hr clock. • Additional experience with visualizations did not change preference for text.	•Older adults more likely to omit time/dose information from the 12-hour clock. •Time, recall, and preference were not influenced by age.	•Icons were unfamiliar •Study did not explore text plus icons •Clock faces outdated • Small sample size Little risk of bias: •Unclear education level of participants
Morrow et al. (46) 1998	Senior citizens College students US (n=72)	Comprehension Time Recall Preference	Time interval visualization added to text	•Timeline •Implied icons of sun/moon • Direct icons of pills	Text alone (“Take...two times each day...take your medicine at 8am in the morning and 8pm in the evening.”)	•Improved comprehension for older and younger participants, especially for complex regimens. •Decreased answer time •Was most preferred. •No influence on recall.	•Comprehension: no interaction of age and format •Time: increased for older adults with visualizations •Recall: marginally improved for younger with visualization	•Participants instructed on the timeline, limiting generalizability • Small sample size Unclear risk of bias: •Unclear education level of participants •Unclear if any missing data •Unclear mode of randomization
Mansoor and Dowse(55) 2003	Outpatient clinic High poverty Low education: from 0–7 yrs. of schooling Low Literacy South Africa (n=60)	Comprehension Preference	Time interval visualization added to text	•Panels •Implied icons of sun/moon & clock faces •Abstract direct icons of medication bottle and dropper filled with liquid medication	Text alone (“Take 1 ml medicine dropper 4 times a day”)	•Improved comprehension of more complex information •Preference for visualizations		•Researcher-conducted interviews could have influenced answers •Limited generalizability due to specific population and their experience with icons Little risk of bias: •Mode of randomization - alternating allocation
Morrow et al.(45) 2004	Congestive heart failure patients Adequate health literacy range US Exp 1. (n=16) Exp 2. (n=32) Exp 3. (n=50)	Preference	Time interval visualization added to text	•Timeline •Implied icons of sun/moon • Direct icons of pills	Text alone (Actual text not included)	•Most preferred visualizations and found that they reinforced knowledge about medication instructions.		•Small study size •Limitations of focus groups used for Experiment 1 No risk of bias
Dowse and Ehlers(56) 2005	Outpatient clinic High poverty Low education LHL: One third with less than a 50% health literacy rating South Africa (n=87)	Comprehension Recall Adherence	Time interval visualization added to text	•Panels •Implied icons of sun/moon, clock faces & person sleeping	Text alone (“Take 2 tablets 2 times a day”)	•Improved comprehension and adherence	•Association between literacy and adherence was highly significant in the control group but was weaker in the experimental group. •Pooling the results indicated that literacy has a significant effect on adherence	•No way to blind participants/researchers •Limited generalizability High risk of bias: •Unclear mode of randomization •Recall phase of 3–5 days •Literacy measured on non-standardized scale
Davis et al.(13) 2009	Outpatient clinic High poverty Mostly female 15% Low literacy, 30% Marginal Literacy US (n=359)	Comprehension	Format for medication timing	• “Time periods” • “Specific times”	“Hourly intervals” “Times per day”	•Use of precise wording on prescription drug labels - best as “time periods” - improved patient comprehension over the use of “hourly intervals” or “times per day”.	•Low and marginal literacy were predictors of poor comprehension •Format interventions did not overcome the increased misinterpretation with low literacy	•Experience with study medications not considered •Subtle differences in word choice and numeric presentation on the instructions •Limited generalizability No risk of bias
King et al.(44) 2011	Adult education program Academic health center High poverty 64% Low Health Literacy (LHL) US (n=45)	Preference	Maximum dosing visualization	•Abstract arbitrary icon of a stop sign • Direct icons of pills	Text Only (Standard over the counter labels)	•Most preferred the version with the visualization.		•Only English-speaking participants •Small sample size •Limited generalizability No risk of bias
Wolf et al.(51) 2011	Outpatient clinic Predominantly African American women English speakers 20% with low literacy 32% with marginal literacy US (n=500)	Comprehension	Format for medication timing Time interval visualization added to text using “time periods”	• “Time periods” •Table	“Times per day” Text only using “time periods” Text only using “times per day”	•Improved comprehension for those with low literacy •Lowered comprehension over text only using “time periods” •Improved comprehension over text only using “times per day”	•Less education was a predictor of lower rates of comprehension. •Patients with low literacy had improved comprehension with “time periods” over “times per day”. •Comprehension of “time periods” was not improved with the added visualization intervention.	•Limited generalizability •Did not have information on patient’s health background Little risk of bias: •Mode of randomization – sequentially given to participants
Yin et al.(14) 2011	Pediatric clinic Academic health center 65% LEP 78% LHL High poverty US (n=299)	Accuracy	Liquid dosing visualization	•Concrete icons of 2 droppers filled with liquid medication	Text only (“1.2mL (0.8 + 0.4 mL)	•Improved accuracy, significantly for those with LHL and limited English proficiency	•Significantly decreased dosing errors for parents with low literacy who received the text-plus-visualization, but not significant for parents with adequate health literacy.	•Limited generalizability No risk of bias
Zargarzadeh et al.(59) 2011	Community and hospital outpatient pharmacies English and Spanish speakers US (n=444)	Preference	Time interval visualization added to text	•Table with time intervals	Text alone (“Take 1 tablet every night”)	•Most preferred the label with the visualization.		•Convenience sample limited generalizability •Most with English as primary language (90%) •Use of hypothetical medications Little risk of bias: •Unclear mode of randomization
Bailey et al.(49) 2012	Outpatient clinic Community based organizations High poverty Limited English Proficiency (LEP) Speak five non-English languages US (n=202)	Comprehension Accuracy Regimen consolidation	Format for medication timing	• “Time periods”	“Times per day”	•Improved comprehension •Improved accuracy •Improved regimen consolidation	•Education was an independent predictor of comprehension and regimen dosing abilities •Positive association between education and comprehension	•Convenience sample limited generalizability •Differences in native languages could have influenced findings if numbers were larger Little risk of bias: •Mode of randomization – random number list created by study team to assign participants
Sahm et al.(47) 2012	Outpatient clinic Academic health center English speakers 30% with LHL Ireland (n=94)	Comprehension	Format for medication timing Time interval visualization added to text using “time periods”	• “Time periods” • “Mealtime anchors” •Table	“Times per day” Text only using “time periods”	• “Time periods” or “mealtime anchors” significantly improved comprehension only for those with limited health literacy •Visualization added to text using “time periods” or “mealtime anchors” improved comprehension for complex regimens with more doses per day	•Older age (at least 60yrs) had lower comprehension levels • “Time periods” or “mealtime anchors” significantly improved comprehension only for those with limited health literacy over “times per day”.	•Small sample size •Use of hypothetical medications •Limited generalizability Little risk of bias: •Unclear mode of randomization
Wallace et al.(48) 2012	Women of child-bearing age in an outpatient clinic LHL: nearly half (49%) with inadequate health High poverty US (n=193)	Preference Comprehension Accuracy	Format for medication timing	• “Specific times”	“Hourly intervals”	53.4% preferred “specific times” No significant effect on comprehension	•Inadequate health literacy skills and low educational attainment had lower comprehension •Educational attainment, HL skills and having a child were assoc. with increased odds of correctly measuring a dose. •Covariates did not have implications on main findings for different numerical formats.	•Limited generalizability Little risk of bias: •Unclear mode of randomization
McCarthy et al.(52) 2013	Patients in emergency department receiving acetaminophen medication Most (72%) had adequate literacy US (n=87)	Accuracy	Format variation for maximum dosing	•Take-Wait-Stop format (Figure 6)	“Take 2 pills every 4 hours. Do not exceed 6 pills in 24 hours”	Improved accuracy	•Non-white race/ethnicity and race/ethnicity “other” more likely to have lower accuracy •No association with age or literacy skills	•Small study •Most (72%) had adequate literacy Little risk of bias: •Unclear mode of randomization
Wolf et al.(50) 2016	Community health center >=30 yrs. of age Diabetes and/or hypertension Taking oral medications English or Spanish speakers 37% with LHL US (n=845)	Accuracy Adherence	Format for timing of medications + Time interval visualization added to text with “time periods”	• “Time periods” + • Table	Text only with “times per day”	Accuracy: •Significant improvements only for English speakers and for English and Spanish speakers on complex regimens Adherence: •Significant improvements only for English speakers and those with limited health literacy	•Time interval visualization added to text using time periods improved accuracy and adherence for those with limited health literacy •Time periods and visualization improved accuracy for Spanish speakers with complex drug regimens	•Community health settings made follow-up difficult. •Very low rates of adherence •Patients may have used different pharmacies •Different literacy measures in Spanish and English. Moderate risk of bias: • Follow up at 3 & 9 months •Only analyzed data from slightly more than half of those recruited
Chan et al.(53) 2017	Outpatient pharmacy Government funded hospital Caregivers of children High poverty LHL Malaysia (n=53)	Accuracy	Liquid dosing visualization	•Concrete icon of dropper filled with liquid medication	Text-only (Dose in mL)	•Improved accuracy		•Limited generalizability •Small sample size Little risk of bias: •Mode of randomization - manual using envelopes
Ng et al.(57) 2017	65 years and older Hong Kong All had basic health literacy Low education (n=50)	Comprehension Preference	Time interval visualization	•Panels •Direct icons of person taking medication •Implied icons of sun & person rising from bed	Text only (In Chinese – “times per day))	•Improved comprehension •Most preferred visualizations	•Lower education levels were associated with poorer comprehension •Higher education levels were associated with better comprehension information in the visualization group.	• Small sample size •Limited generalizability Little risk of bias: •Unclear mode of randomization
Yin et al.(43) 2017	Pediatric clinic Academic health center English and Spanish speakers Parents and guardians (28% had LHL) US (n=493)	Accuracy	Liquid dosing visualization	•Concrete icon of dropper filled with liquid medication	Text only (Dose in “mL” or “mL/tsp”)	•Improved accuracy with significantly less risk of large overdosing errors.	•No impact of language on overall risk of making errors •No interaction of health literacy was significant for text vs text with pictograms	•Limited generalizability No risk of bias
Leiner et al(54) 2018	Pediatric clinic Academic health center Parents and caregivers Mostly women Mostly Spanish speakers Low literacy Low education US (n=359)	Comprehension Recall	Time interval visualization	•Panels •Implied icons of sun/moon and person looking at watch with an hour interval displayed •Abstract direct icons of medicine containers, dispensers, and medication	Text only “Times per day”, “Hourly intervals”	•Improved comprehension and recall		•Limited generalizability Moderate risk of bias: •Readers were asked to answer questions that were not listed • Mode of randomization - sequential
Phimarn et al.(58) 2019	Hospital & primary care LHL Taking medication Low income Thailand (n=134)	Comprehension Adherence Preference	Time interval visualization	•Panels •Implied icons of sun/moon, clock faces & activities associated with time of day • Direct icons of pills	Text only (“traditional labels”)	•Slightly improved comprehension and adherence •Most preferred visualizations		•Limited generalizability •Pharmacists counseled patients on pictograms use Moderate risk of bias: • Mode of randomization – manual •Follow-up phase at 14 days

Outcomes were varied. The majority (14 papers) measured comprehension, 8 preference, 7 accuracy, 5 recall, and 3 adherence (some measured more than one). Comprehension was measured differently: qualitatively analyzed responses (13), search tasks and inference questions (46), and asking participants to describe how to take medications or conceptualize how many pills to take (46, 47). Accuracy was also measured in multiple ways; by asking questions on the measurement of dose and frequency (48), comparing weights of measured doses to a reference weight (14), and asking participants to physically demonstrate dose and frequency (49).

3.2. Format interventions

Seven studies evaluated the use of different format interventions to express numeric information through numerals and words (13, 47–52). We identified different categories of format interventions.

Take-Wait-Stop Format (Figure 6 and Table 5)

Figure 6.

Take-Wait-Stop format

One paper by McCarthy found that a unique Take-Wait-Stop format reduced errors associated with exceeding the maximum daily dose by 2.5 times compared to the standard label (52).

Medication timing formats (Tables 4 and 5)

Table 4.

Outcome matrix of format interventions for medication timing, in chronological order

Author Year	Outcome	Time periods Take 2 tablets in the morning and 2 tablets at bedtime	Specific times Take 2 tablets at 8 A.M and 2 tablets at 5 PM	Mealtime anchors Take 2 tablets with breakfast and 2 tablets with dinner	Hourly intervals Take 2 tablets every 12 hours	Times per day (Arabic numerals) Take 2 tablets twice daily	Times per day (numeric words) Take two tablets twice daily
Davis et al. 2009 (13)	Comprehension	Better (than hourly intervals or times per day)	Better (than hourly intervals or times per day)	-	Worse (than time periods or specific times)	Worse (than time periods or specific times)	-
Wolf et al. 2011 (51)	Comprehension	Better (than times per day)	-	-	-	Worse (than time periods)	-
Bailey et al. 2012 (49)	Comprehension Accuracy Regimen consolidation	Better (than times per day)	-	-	-	-	Worse (than time periods)
Sahm et al. 2012 (47)	Comprehension	Better (than times per day)	-	Better (than times per day)	-	-	Worse (than time periods or mealtime anchors)
Wallace et al. 2012 (48)	Preference Comprehension	-	Better (than hourly intervals) Preference only	-	Worse (than specific times) Preference only		-
Wolf et al. 2016 (50)	Accuracy Adherence	Better (than times per day)	-	-	-	Worse (than time periods)	-

Five studies (13, 47, 49–51) evaluated a format that used explicit instructions for the timing of medication dosing to simplify medication administration instructions with the goal of increasing patient understanding and adherence, and improving health outcomes (15, 42). These recommendations include the use of time periods (morning, noon, evening, and bedtime). Five of our included studies confirmed that the use of time periods (when compared to times per day) improved outcomes including comprehension (n=4)(13, 47, 49–51), accuracy (n=2)(49, 50), regimen consolidation (n=1)(49) and adherence (n=1)(50).

Davis’s study evaluated the use of “specific times” and found that patients were more likely to comprehend instructions with either “specific times” or “time periods” when compared to “hourly intervals” or “times per day” (13). A study by Wallace found a user preference for specific times when compared to “hourly intervals” but identified no change in comprehension (48). Sahm found that using “mealtime anchors” (timing the medication to coincide with mealtimes) improved comprehension when compared to using “times per day” (47).

3.3. Visualization interventions

Nineteen studies evaluated visualization interventions (Table 5).

1. Liquid dosing visualization

Five studies evaluated liquid dosing visualizations, four with caregivers of pediatric patients, and one with adults (14, 43, 53–55). Three of these used concrete icons of syringes to demonstrate the fill line for the correct dose (14, 43, 53). The other two included abstract, direct icon images of droppers filled to the appropriate line (54, 55). A 2011study by Yin used concrete icons to visualize how to use one dropper two times to achieve the correct dose (14). All these studies found improved outcomes with the addition of the liquid dosing visualizations compared to text only. These metrics include accuracy (n=3), comprehension (n=2), preference (n=1), and recall (n=1). These studies provide strong evidence for the use of liquid dosing visualizations.

2. Maximum dosing visualization

One paper by King found a majority preference for a maximum dosing visualization to prevent overdosing of medication. This visualization consists of an arbitrary icon recognizable as a stop sign image with text communicating the maximum daily dose (44) (Figure 4).

3. Time interval visualization

Time interval visualizations were the most common type of visualizations (n=13). The concept of time and time periods were often represented by icons, most frequently by the sun and moon (n=8). Other icons included a clock (n=3), light and darkness (n=3), person sleeping in a bed (n=2), person waking from a bed with a sun icon (n=1), a watch with the period of time expanded in view (n=1) and eating a meal while a rooster is crowing (n=1).

Most of these studies found that the addition of time interval visualizations resulted in one or more improved communication outcomes (n=8) (45, 50, 54–59). However, for an additional three studies, the benefit occurred only when explanatory text was added to the visualization (46, 47, 51). Two studies from Morrell and Morrow found that their visualizations without explanatory text actually reduced comprehension (60, 61). There were four different types of organizing structures identified in our studies: timelines, tables, and panels.

a. Timelines

Three types of timelines (a linear timeline, a 24-hour clock, and a 12-hour clock) were used in papers by Morrow (45, 46, 61). The author’s 1996 study found that text-only labels resulted in better outcomes than timelines alone. Participants found the unfamiliar 12-hour clock to be particularly confusing. Additional experience with these icons did not result in better outcomes (61). In Morrow’s second paper, the addition of explanatory text improved the performance of a linear timeline (46). In Morrow’s third paper, the linear timeline with explanatory text was preferred over the version with only the explanatory text (45). Weak quality of evidence plus heterogeneity of the timelines precludes any evidence-based recommendations for the use of timelines.

b. Tables

The tables used in four of the included studies expressed time with words and numbers in an organizing structure and leveraged natural mappings of left-to-right (n=3) or up-to-down (n=1) to express the progress of time from past to future (47, 50, 51, 59). Three used time periods such as morning, noon, evening and bedtime (47, 50, 51), two used specific times, (50, 59), and one used both (50). Studies by Sahm and Wolf, 2011 found that the addition of a table to a label with explanatory text decreased comprehension (47, 51). A subsequent 2016 paper by Wolf used a modified version of the table which included time periods and actual times with associated explanatory text and found a modest improvement in accuracy over the standard text without a table (50).

c. Panels

Multiple studies used panels in their visual interventions (54–58, 60). Most of these visualizations incorporated icons into panels that leveraged natural mappings of left-to-right to express the progress of time. However, not all studies used this spatial pattern. The study by Morell in 1990 brought together several different communication elements in close proximity, without visualizing time (60). All these studies, except for the study by Morell, found improved outcomes with the use of visual interventions compared to standard text labels. Leiner’s study added images in sequential patterns in panels based on a ‘graphic narrative’ style similar to a comic book and found improvements in comprehension and recall compared to written text instructions (54).

3.4. Covariate findings

3.4.1. Literacy

Two studies compared labels with visualization interventions to labels with text only for participants with low literacy. Both found improvements in outcomes (54, 56). Leiner found increased recall and comprehension after including a visualization in the label and Dowse found less of an association between literacy and adherence with the visualization. However, the moderate and high risks of bias for these two studies plus the heterogeneity of the visualizations produces weak evidence for the use of visualizations for those with low literacy.

3.4.2. Health literacy

Of the six studies that assessed health literacy as an independent predictor of outcomes, three studies by Davis, Wallace and Yin 2011 found significantly worse outcome measures among those with low health literacy (13, 14, 48), and studies by McCarthy, Sahm and Wolf 2011 did not (47, 51, 52).

All but one of the eight studies that evaluated the effect of a visualization intervention on those with low health literacy (LHL) identified improvements in at least one outcome metric (14, 43, 44, 47, 50, 53, 58). Chan, King, Phimarn and Yin’s studies took place in populations with known low health literacy. Chan found improvements in accuracy (53), Phimarn found improved comprehension and adherence (58), King found a preference for the label with the visualization (44) and Yin found increased accuracy (14).

Sahm’s study evaluated the use of a visualization intervention along with a format intervention and found that those with low health literacy were the only ones to show an improvement in comprehension (47). Yin 2017 found that the overall improvement in dosing errors with a liquid dosing visualization did not vary with levels of health literacy (43). In 2011, Wolf’s study looked at patient-centered labels (PCL) and determined that the addition of a table did not improve comprehension for those with LHL compared with the PCL alone (51). However, Wolf’s 2016 study found increased adherence for patients with LHL after inclusion of a PCL label that included a table (50). These studies generate strong evidence for using visualizations for a population with LHL; however, the heterogeneity of the visualizations lowered the level of this recommendation.

Two studies evaluated the effect of using format interventions for those with LHL. Wallace determined that the format intervention did not predict outcomes in this group (48). Davis’s study found that even with the improved comprehension associated with a PCL, those with LHL still had worse outcomes. These authors felt that patient counseling could further address health literacy deficits (13).

3.4.3. English proficiency

Yin’s 2011 study found that health literacy and English proficiency were closely related and after controlling for randomization status and health literacy, they found that limited English proficiency (LEP) was not significantly related to dosing errors.

Three studies examined the impact of visualization interventions on those with LEP. Yin’s 2011 study found that the use of visualization interventions increased accuracy only for those participants with LEP (14). However, in 2017, Yin found that when labels were available in both English and Spanish, there was no interaction with the use of liquid dosing visualizations and medication errors for those with LEP (43). Leiner’s study found that comprehension improved for those with LEP when adding visualization interventions in labels available in both English and Spanish (54). These studies demonstrate strong evidence for the use of visualizations for populations with LEP. However, the heterogeneity of the visualizations lowered the level of recommendation.

Studies by Bailey and Wolf looked for associations between participants with LEP and the effects of format interventions (49, 50). Bailey found that presenting instructions using time periods in translated labels resulted in improved comprehension, accuracy, and medication consolidation for this population (49). However, Wolf 2016 found that the use of time periods in translated labels resulted in improvements in accuracy and adherence only for those with adequate English proficiency (50).

3.4.4. Predominantly Non-English-speaking international populations

All five of the non-English speaking international studies that evaluated visualization interventions versus text-only demonstrated improved outcomes, including comprehension (55–58), preference (55–58), accuracy (53) and adherence (56, 58). Studies by Dowse, Mansoor, Ng and Phimarn used USP validated images. Phimarn’s images were further adapted and validated specifically for the Thai culture (57, 58). Mansoor and Dowse used USP guidelines while developing their visual interventions, which were tested in the South African population. These studies provide strong evidence for the use of visualization interventions with validated icons; however, the icon heterogeneity lowered the level of recommendation (55).

3.5. Strength of evidence

Many of the included studies had small sample sizes and limited generalizability. The top four reasons for bias concerns were: 1) non-standard or unclear methods of randomization; 2) possible protocol deviations due to long follow-up phases of the study that leave results open to outside influences; 3) no validated or established scale used to measure covariates such as health literacy or numeracy; and 4) high rates of missing data or lack of clarity about missing data (Table 5). Most of our papers had little or no risk of bias; however, four papers had moderate or high risks of bias (Table 6). See Tables 8 and 9 in Appendix A for concern and risk of bias criteria. (62).

Table 6.

Risk of bias assessment

Level of risk of bias	Criteria	Papers
None	No identified concerns	n=5 (13, 14, 43–45)
Little	One medium level of concern or one area of unclear information	n=11 (47–49, 51–53, 55, 57, 59–61)
Moderate	Two medium levels of concerns	n=3 (50, 54, 58)
High	More than two medium levels of concerns	n=1 (56)
Unclear	More than one area of unclear information	n=1 (46)

3.6. Evidence-based recommendations

Based on findings of related studies, we generated recommendations to improve medication instructions (Table 7). The risk of bias scores of the relevant papers were combined with the consistency of the outcomes to generate the overall strength of evidence for each recommendation. For example, we looked at a group of studies evaluating the use of visualizations in liquid medication instructions. Since these studies had little or no risk of bias and all found positive impacts on outcome measures (high consistency), we determined this to be strong evidence. Using the criteria outlined in Table 1, we identified six recommendations with strong evidence.

Table 7.

Evidence-based recommendations for medication instructions

Format interventions
Level of recommendation	Recommendation	Strength of evidence	Homogeneity of interventions	Comments
I	Text instructions should use “time periods” to depict frequency instead of “times per day” (13, 47, 49–51).	Strong	High	These results support the IOM’s Universal Medication Schedule, as well as other established guidelines (15, 17, 24, 25).
II			N/A
	Text instructions should use “mealtime anchors” to depict frequency instead of “times per day” (47).	Weak (1 study)	One study	One study found improved comprehension compared to times per day. Further studies could compare to “time periods.”
III	Text instructions should use specific “times of day” to depict frequency instead of “times per day” or “hourly intervals” (13, 48).	Weak	High	One study showed an improvement in comprehension. A second found a preference for specific times, with no improvement in comprehension.
	Text instructions should use the “Take-Wait-Stop” format to communicate maximum daily dosing limits (52).	Weak (1 study)	One study	One study found improved accuracy with the format depicted in Figure 6.
Visualization interventions
Level of recommendation	Recommendation	Strength of evidence	Homogeneity of interventions	Comments
I	Liquid medications instructions should use visualizations of liquid medication devices to depict specific doses (14, 43, 53–55).	Strong	High	There are variations in the level of concreteness of the icons used to visualize liquid medications. Further studies could identify optimal icons.
II	Medication instructions should use visualizations tables with associated explanatory text (47, 50, 51, 59).	Moderate	High	All studies found improved outcomes with a table plus associated explanatory text. Two of these found that the use of a table alone (without explanation) reduced comprehension outcomes.
	Medication instructions should use visualization panels (54–58, 60).	Moderate	High	These panels generally progressed spatially from left to right and depicted a variety of information. Two studies with moderate and one with high risks of bias.
	Medication instructions should use visualizations for populations with low health literacy (14, 43, 44, 47, 50, 51, 53, 58).	Strong	Low	All studies showed an improved outcome for those with LHL. Of the 3 studies that looked for an association between LHL and the intervention, 2 found that the improvement varied with the LHL, and one did not.
	Medication instructions should use visualizations for populations with limited English proficiency (14, 43, 54).	Strong	Low	In one study, the difference in better outcomes for those with LEP disappeared after the instructions were translated into the preferred language.
	Medication instructions should use visualizations for non-English speaking populations (53, 55–58).	Strong	Low	These visualizations included panels and liquid medication visualizations.
	Medication instructions should use images that have been validated (52, 55–57).	Strong	Low	All studies used USP pictograms, although most adapted them to the user population.
III	Medication instructions should use visualizations for populations with low literacy (54, 56).	Weak	Low	The evidence was limited to two studies with positive findings, however, with moderate and high risks of bias.
	Maximum dosing visualizations may be preferred to words/numbers alone (44)	Weak (1 study)	One study	This study found a preference for instructions with this visualization. However, no evidence is available about comprehension-related outcomes.

Using the criteria outlined in Table 2, we developed an ordered list of evidence-based recommendations based on the strength and homogeneity of the related group of studies. The recommendations in Level I had strong evidence and high homogeneity of interventions. These were to use ‘time periods’ instead of ‘times per day’ and to use visualizations of medication devices for liquid medication instructions (Table 7).

4. DISCUSSION AND CONCLUSION

4.1. Discussion

Our included studies used variable vocabulary, types of visualizations, study methods, and outcome measures. To enable effective comparisons, we developed a novel means of analysis using a concept map to parse out concepts, elements, and structures within each study. We combined risk of bias with the consistency of outcome findings to determine the strength of evidence. We then incorporated the homogeneity of the interventions with the strength of evidence to determine the level of evidence-based recommendations, graded from Level I to Level III.

Using these novel techniques, our review has generated three levels of evidence-based recommendations to improve comprehension-related outcomes for quantitative information in medication instructions. Our findings confirm some previous guidance statements with peer-reviewed evidence; however, they suggest the need for updates to other guidance statements and identify gaps not addressed.

Established medication instruction guidelines are generally based on expert opinions and limited empirical research. Our review has generated recommendations listed in Table 7 that we have compared to current guidelines from the following sources: Institute of Medicine, (IOM), Institute for Safe Medication Practices (ISM), United Sates Pharmacopeia Convention (USP), National Council for Prescription Drug Programs (NCDPD), the Australian Commission on Safety and Quality in Health Care (ACSAQHS) and an international expert panel represented by Raynor and Dickinson (15, 17, 21, 24, 25, 42).

The Level I format recommendation in our table is to use ‘time periods’ instead of ‘times per day’. This supports the UMS rule of explicit instructions and is widely supported in our included guidelines that address timing formats (15, 17, 18, 20, 25). The goal of these guidelines is to simplify medication instructions to improve comprehension and adherence, thereby improving outcomes.

Our review also generated three Level III format recommendations that have been mentioned sporadically in our included guidelines. The use of ‘mealtime anchors’ supports IOM guidelines as a UMS potential standard. The use of ‘specific time frames’ supports a helpful strategy listed in ACSQHC, with the caveat that one must consider the appropriateness for different users, such as shift workers who may prefer different times (15, 17). The ‘Take-Wait-Stop’ format supports guidance from the ACSQHC for maximum dosing communication (17, 52).

Our review generated one Level I visualization recommendation to use visualizations for liquid medication instructions, using icons representing dosing instruments and medication amounts to depict quantities (14, 43, 53–55). This recommendation was not found in our included guidelines and should be considered in future guideline updates. Liquid dosing visualizations are useful in situations where the liquid doses are considered the most cognitively challenging portion of the instruction. Although there was general homogeneity in the liquid dosing icons used, there were differences in the level of concreteness of the icons.

Our review generated additional Level II and III recommendations for visualization interventions, including the use of organizing structures, such as tables and panels. This type of intervention is not included in many guidelines. However, the recommendation to use tables with associated explanatory text is supported by the ACSQHC in the context of complex medication needs (17). In addition, the recommendation to use images that have been validated, such as the widely adopted USP images, is supported by many accepted guidelines (15, 17, 21, 24, 25, 42, 63). We also generated a Level III recommendation to use a ‘maximum dosing visualization’ that has not been included in these guidelines.

We explored the differential impact of visualization interventions on patients with low literacy, LHL and LEP. The recommendation to use visualizations to improve communication of medication instruction for low literacy populations was Level III, due to limited numbers of studies done with this population. This recommendation is supported by the NCPDP, and the International Pharmaceutical Federation which supports the use of “pictograms” to give health care providers a way to communicate medication instructions to people without a common language or without competent literacy (10, 42, 64).

Our covariate findings support the use of visualization interventions for populations with limited health literacy and English proficiency. Since it is not always possible to identify these limitations in a given population, strategies to improve comprehension should be used universally. This would comply with universal precautions suggested by the Agency for Healthcare Research and Quality in 2010 to lower the risk of misunderstanding medication instructions and medication errors (65, 66). Visualizations improved outcomes for patients with LEP but did not consistently provide additional benefits when instructions were provided in the patients’ preferred language. Therefore, the preferred approach is to translate into the preferred language, as recommended by the USP and NCDPD (25, 42).

One of our included studies found that despite improvements in outcomes for those with low literacy after the use of more explicit language, this group still had worse overall comprehension. More consistent stratification of findings in future studies could help identify additional strategies to improve comprehension of medication instruction for vulnerable groups (13).

We examined the impact of visualizations on instructions given to non-English speaking international populations. There was strong evidence to support this type of intervention but due to heterogeneity of the interventions, these studies generated a Level II recommendation. There was strong evidence that using validated images improved outcomes, and it’s possible that there are additional benefits to adapting and validating images to the specific populations. This approach would incorporate the ecological and cultural environments which influence the ability of users to draw meaning form particular images (58).

4.2. Limitations

Since this review is a subset of a larger review, the search terms used originally were broad, retrieving a very large group of articles that had to be manually sorted. It is possible that either the search terms failed to retrieve relevant articles, or that the manual reviewers failed to identify them in the large group. We studied only written instructions with modifications to the numerical information, excluding studies of modifications that did not directly address numbers, as well as studies of oral communication, teach-back techniques, and other interventions that might improve comprehension. We also did not address other non-numerical aspects that are not primarily quantitative but may influence timing. There may be other strategies that we did not identify if they were no direct comparison studies.

Some of the studies in our review dated back to the 1990s, which might contain methods and images that are less relevant now. Many of our studies had small sample sizes and limited generalizability. There were varying levels of risk of bias, with nearly 40% having some limitations in study design, data collection, or analysis that could weaken confidence in their findings. However, we have integrated limitations into our system of strength determination. We have also factored in consistency to assess reproducibility. Based on this system, Level I recommendations are supported by studies with low levels of limitations and high homogeneity, such as the use of “time periods”.

4.3. Conclusion

The effective communication of numerical information in medication instruction is essential to minimize medication errors. This is particularly important for populations at high risk for medication errors, including caregivers of pediatric patients, and those with low literacy, health literacy, and LEP.

In this review, we have used novel study analysis techniques and developed criteria to determine evidence-based recommendations. We have determined that for liquid medications, illustrating doses with icons was more effective than stating the dose in words and numbers alone. Organizing structures of visualization with moderate recommendations include the use of panels and tables with explanatory text and images that have been widely validated, and possibly adapted to specific populations.

Also, for any type of medication, instructions that specified the time periods to take a medication were more effective than instructions that specified times per day, which complies with UMS guidelines.

The positive impact of using visualizations for populations with limited health literacy and English proficiency supports their increased use for all populations. This “universal precautions” approach would improve outcomes for patients with unidentified limitations.

4.4. Practical implications

For liquid medications, prescribers should illustrate doses with markings on syringe icons rather than just stating the dose in words and numbers. For any type of medication, prescribers should specify the time periods to take a medication (morning, noon, night, or mealtimes) rather than pills per day.

Although some of this literature is relatively old, these lessons are not yet widely known among healthcare providers, so integrating these findings into provider training would help translate them into practice.

Our recommendations could be useful for healthcare providers, medical institutions, software vendors, informaticists, patient advocates, and policy makers. We have highlighted areas that could benefit from additional research, especially to improve outcomes for those with limited health and numeracy literacy.

Future research could explore strategies to further reduce the communication gap for vulnerable populations. The categorization, terminology, and recommendation criteria we have developed for this systematic review could be utilized in future studies when comparing and categorizing different intervention types. Future research could explore different forms of multimedia to improve communication with endpoints including healthcare outcomes and expenditures.

Highlights.

• A novel concept map can be used to classify quantitative medication instructions

• Using visualized liquid medication doses improves comprehension of instructions

• Time period-based instructions improves comprehension of medication instructions

• Medication instruction images may improve outcomes for limited health literacy

• Using validated images in medication instructions may improve outcomes

Appendix A

Ovid MEDLINE (ALL – 1946 to January 7, 2019)

Searched on January 9, 2019

No language, publication date, or study type restrictions

Line #	Search
1	Multimedia/
2	(animation or multimedia or multimedium).tw.
3	Audiovisual Aids/ or Webcasts as Topic/
4	(Audio* or webcast* or podcast* or RSS or Really Simple Syndication).tw.
5	Videotape Recording/ or Video Games/ or Video Recording/
6	(video or videos or videotape or videorecording or computer game* or visualization).tw.
7	Computer Graphics/ or Medical Illustration/ or Decision Trees/
8	(bar chart* or pie chart* or drawing* or graphic* or graph or graphs or picture* or pictorial representation or pictogram or pictograph or illustration* or ornamentation or imprints or infographic or infogram or histogram or diagram or diagrams or icon or icons or visual representation* or data format or data presentation or risk communication).tw.
9	Anxiety/
10	(anxiety or anxious or anxieties or nervousness or fear or concern or apprehension or worry).tw.
11	Comprehension/
12	(comprehension or comprehending or understanding or readability).tw.
13	Decision Making/ or Choice Behavior/
14	(decision making or decision satisfaction or decisional conflict or decisions or judgement or choice behavior or choice behaviour).tw.
15	Medication Adherence/
16	((medication or drug or dose or dosing or dosage) adj2 (adherence or nonadherence or noncompliance or non-adherence or persistence or compliance or non-compliance)).tw.
17	Medication Errors/
18	((medication or drug or dose or dosing or dosage) ad)2 (error or errors)).tw.
19	(perceived effectiveness or perceived efficacy or perceived risk or perceived susceptibility or perceived usefulness).tw.
20	Mental Recall/
21	recall.tw.
22	Trust/
23	(trust or distrust or mistrust or trustworthiness).tw.
24	Emotions/
25	(emotional response or emotional factor or expressed emotion).tw.
26	Mathematical Concepts/
27	(mathematical concept* or numeracy or numeric or numbers or numeral or numerical or numerosity or quantitative data or quantitative information or quantitative literacy or statistical information or statistical literacy or statistical interpretation or statistical data or natural frequency or natural frequencies or risk comprehension or risk interpretation or risk reduction).tw.
28	26 or 27
29	or/1–8
30	or/9–25
31	28 and 29 and 30

Table 8.

Criteria for bias concerns

Bias concerns	Low	Medium	High	Not applicable	Unclear
Are there bias concerns about the sample selection?	If general population, then less than 50% with college degrees or <65% in high school plus some college (or more).	If highly educated (undergraduates, >50% with college degree, or >65% with high school plus some college) without deliberate stratification.	If participants have extreme specialized knowledge (e.g., designers of the presentation format).		If the sample is not described.
Are there concerns related to the quality of the randomization process?	If generated by computer, or a well-established statistical method.	If manually randomized (e.g., hand-shuffled envelopes) or mode of randomization is unclear.	If should be randomized but is not effectively randomized.	If randomization is not needed.	If unclear whether the trial is randomized.
Are there concerns related to protocol deviations or concurrent events that could have affected outcomes?	If there are short experiments with no follow-up unless something seems questionable.	If a recall phase was present in the study.	If there is a stated deviation and/or revision to method partway through study.		If nothing can be determined about the experiment length or follow up.
Are there concerns related to the validity of relevant outcome measures?	If a previously validated or well-established measure, or a coding scheme with adequate information on the measure, or a self-reported measure with good face validity was used.	If a coding scheme does not have adequate information on the measure.	If the measure is a clear misrepresentation of the construct it claims to measure.		If a measure is not described in enough detail to assess validity.
Are there concerns related to the validity of how literacy, health literacy, numeracy, health numeracy and/or graph numeracy were measured?	If relevant covariates are measured and coding schemes include adequate information on the measures.	If relevant covariates are measured and the coding scheme does not include adequate information on the measure, or a previously validated scale is modified	If there are highly biased determinations (e.g., subjectively assessed by study personnel).	If no relevant covariates are measured.	If the covariate measures are not described in enough detail.
Is there concern related to how missing data is handled?	If there is no missing data, or only a small amount of missing data (approx. < 5%), or if information related to missing data is reported and is non-differential.	If there is a large amount of missing data (approx. >5%), or if missing data appeared differentially, are relevant to the study purposes and are not accounted for by the researchers.	If there is highly inappropriate handling of missing data (e.g., guessing what the participant would have said).	If the results have no missing data.	If there is insufficient information to determine if any data is missing.

Table 9.

Criteria for overall risk of bias

Risk of bias	Bias concerns identified by Table 8 criteria
Little	1 medium level concern, or 1 unclear area of concern
Moderate	2 medium level concerns
High	More than 2 medium level concerns, or at least 1 high level concern
Unclear	More than 1 unclear area of concern