Effects of a methodological infographic on research participants’ knowledge, transparency, and trust

Abstract

Objective

Given participants’ research literacy is essential for clinical trial participation, evidence-based strategies are needed that improve literacy and easily accessed online. We tested whether an infographic letter—that illustrated how dropouts can distort study conclusions—improved participant knowledge about the impact of dropouts relative to a control letter.

Methods

In three distinct online samples purposely recruited to assess reproducibility, young ethnically diverse adults were randomized to read an infographic letter or control letter in a hypothetical scenario. Secondary outcomes included participants’ perceived transparency of the research organization, perceived value of retention, and perceived trust of the organization. We purposely included two discriminant items, perceived value for the trial outcome and keeping commitments in general, both hypothesized not to change.

Results

Across samples, ~20% more infographic participants correctly answered how dropouts affected study conclusions than control participants. For example (Experiment 3), nearly 90% of infographic participants correctly answered versus only two-thirds of controls (88.7% versus 66.7%, absolute percentage difference 22.0%, p < .0001). Infographic participants had substantially higher transparency, perceived value for retention, and trust (Cohen’s ds = 0.4–1.0, ps < .0001), yet importantly did not value the study outcome or report keeping commitments more than control participants (Cohen’s ds = 0.0–0.1, ps > .10).

Conclusions

Promisingly, this transparent, visually powerful methodological infographic improved knowledge and trust. Future trials could embed and experimentally test whether such low-cost online infographics improve not only research literacy, but also trial retention, especially among populations with less initial trust about research.

Ensuring participants’ research literacy is essential for informed participation in clinical trials, including for individuals less likely to participate in clinical trials such as young adults or underrepresented racial/ethnic individuals (Goldberg & Kiernan, 2005; Oh et al., 2015; Powell et al., 2017). In prior work, we described a novel practical approach for improving participants’ research literacy—the use of interactive in-person group-based orientation sessions held prior to randomization (Goldberg & Kiernan, 2005). We designed these sessions in part to clearly explain the methodological rationale for trial procedures, such as completing all follow-up assessments, using easy-to-understand graphic images that avoided intimidating scientific jargon. We also speculated that transparency about methodology could increase participant trust. Given most clinical trials cannot hold in-person orientation sessions, it is critical to develop and test evidence-based strategies that improve literacy, yet are easily accessible via online formats.

In the current study, we experimentally tested whether a 1-page infographic letter—that illustrated how dropouts can distort study conclusions—improved participant knowledge about the impact of dropouts relative to a control letter. Graphic images are rarely evaluated for comprehension before use (Tufte, 1983). Poor trial retention undermines external validity, especially when participants unresponsive to an intervention or dissatisfied with the experience drop out. We designed the infographic to convey that a ‘true picture’ of trial outcomes is preferred scientifically even if the trial does not ‘work’ and/or individual participants feel unsuccessful (Goldberg & Kiernan, 2005). This is critical in behavioral medicine as trial participants are typically not masked to behavioral interventions and are aware whether they made recommended changes in the targeted health behaviors or achieved objective outcomes such as weight loss.

Here, in three distinct online samples purposely recruited to assess reproducibility, ethnically diverse young adults were randomized to read an infographic letter or control letter in a hypothetical scenario. Secondary outcomes included participants’ perceived transparency of the research organization, perceived value of retention, and perceived trust of the research organization. We purposely included two discriminant items that assessed conceptually-related potential confounds (Kiernan, 2012), perceived value for the trial outcome and keeping commitments in general, both hypothesized not to change.

Methods

In Experiment 1, individuals in an online research pool at a private liberal arts university received research credit for participation. In Experiment 2, community college students in the Stanford Research Experience Program research pool, a collaboration between Stanford University and local community colleges, received research credit for participation. In Experiment 3, Mechanical Turk online workers ≥18 years and approved on ≥ 95% of ≥50 past studies received $3.50 for the estimated 30-minute activity. All three experiments were exempt protocols. The Santa Clara University Institutional Review Board (IRB) approved Experiment 1 and the Stanford University School of Medicine IRB approved Experiments 2 and 3. In all three online survey experiments, participants read an information page and agreed to participate before starting, were randomly assigned to read one of two letters, and then answered survey questions.

Letter conditions

Using identical language, both letters asked participants to imagine a hypothetical research organization was evaluating an online program to increase fruit and vegetable intake; they were enrolled in the program and had received a letter from the research organization containing $20 for already completing a baseline survey; and the letter informed them that completing the remaining follow-up surveys would result in an additional $75.

The 1-page control letter only contained the information above. The 1-page infographic letter also included two visually powerful graphic images and simple text concisely explaining the methodological impact of dropouts on conclusions, i.e., a study could look more successful than it really was, and the rationale for high retention rates, i.e., a ‘true picture’ of trial outcomes is preferred by researchers even if they learn the trial was not helpful for some people (Supplemental Figure 1). The infographic was guided by two statistical graphic design principles: small multiples (an image repeated several times to invoke explicit comparisons across images), and a streamlined design without unnecessary details to maximize the data-ink ratio (Tufte, 1983).

All experiments had a 1:1 allocation ratio. In Experiments 1 and 2, participants were allocated to letter condition based on the last digit of their (randomly generated) university research pool ID number. The first author randomly assigned the overall allocation sequence (which 0–9 digits assigned to which condition). In Experiment 3, participants were randomly assigned to letter condition via Qualtrics software in block size of two. Investigators were masked to condition and outcomes until a particular experiment was completed.

Measurements

Demographics

Participants indicated gender, race/ethnicity, age, and educational level.

Primary outcome

Knowledge was assessed with the hypothetically-phrased question below (the first answer was the correct response).

Imagine people are in a new online program to help them eat more fruits and vegetables. They are asked to fill out a survey at the end of a program so the evaluation team can see if the program worked.

Now imagine that the people who did not eat more fruits and vegetables by the end of the program did not fill out their surveys. How would that affect what the evaluation team would conclude about the program?

• The program would look more successful than it really was

• The program would look less successful than it really was

• It would have no effect

Secondary outcomes

Transparency of the research organization, perceived value of retention, and trust of the research organization were assessed with 5-point scales scored 1–5 with the following response options (Not at all, A little bit, Moderately, Quite a bit, Extremely). We assessed value of retention rather than behavioral intention, given the hypothetical scenario.

• To what extent do you think this organization is willing to find out that the online program may not be helpful for people?

• To what extent do you trust the organization that sent this letter?

• It is important to me that I complete the survey at the end of the program, whether or not I eat more fruits and vegetables

Discriminant items

In all experiments, perceived value of study outcome and keeping commitments were assessed with the same 5-point scales and response options as above.

• It is important to me that I eat more fruits and vegetables

• It is important to me to keep the commitments I make

Statistical power

In Experiment 1, and in the first test of the infographic letter, we did not have an a priori effect size estimate for knowledge, thus all interested participants were randomized. In Experiments 2 and 3, we wanted to detect an effect size for knowledge as large as Experiment 1 (absolute percentage difference of ≥15% assuming 75% of controls correctly answered), and a Cohen’s d as small as 0.3 on secondary outcomes. To detect d = 0.3 with a two-sided t-test, 5% significance level, and 80% power, we needed at least 176 per condition. In Experiment 2, all interested students were randomized. In Experiment 3, sample size was pre-specified via online software. Using SAS 9.4, the primary outcome was the percentage selecting the correct response (program more successful than it was), analyzed with a chi-square test and 95% confidence intervals (CI). Remaining outcomes were continuous, analyzed with t-tests and 95% CIs.

Results

In all experiments, there was no differential dropout by letter condition. In Experiment 1, 92% (77/84) of participants were included in the primary outcome analysis (7 dropped out prior to the primary outcome question). In Experiment 2, 99% (406/411) were included (5 dropped out). In Experiment 3, 100% (n=351) of participants were included in the primary analysis (13 workers took it twice, but post hoc analyses with and without them revealed identical results).

Demographics

In all experiments, over half were women (Table 1). In Experiments 1 and 2, ≥60% of participants were from racial/ethnic minority backgrounds and 20% were from multi-racial/ethnic backgrounds. In Experiment 3, and as designed to extend generalizability, more than half did not have a bachelor’s degree and over 20% of participants were ≥40 years.

Table 1.

Sample size and demographic characteristics, Experiments 1–3

	Experiment 1		Experiment 2		Experiment 3
Variables	Liberal arts university*		Community college†		Mechanical Turk
	%	n	%	n	%	n
Sample size		77		406		351
Demographic characteristics
Women	71.4%	55	68.7%	279	46.7%	164
Race/ethnicity
American Indian/Alaskan Native	0.0%	0	0.0%	0	0.0%	0
Asian	20.8%	16	34.2%	139	8.3%	29
Black/African American	1.3%	1	2.2%	9	5.7%	20
Hispanic or Latino	3.9%	3	15.3%	62	3.4%	12
Native Hawaiian/Other Pacific Islander	-	-	1.7%	7	0.0%	0
White	40.3%	31	26.9%	109	70.4%	247
Multi-Racial/Ethnic	33.8%	26	19.7%	80	12.3%	43
Age category
≤ 21 years old	93.5%	72	46.3%	188	5.7%	20
22–29 years old	6.5%	5	53.7%	218	40.7%	143
30–40 years old	-	-	-	-	29.6%	104
40–49 years old	-	-	-	-	12.8%	45
≥ 50 years old	-	-	-	-	11.1%	39
Education category
High school degree or less	-	-	-	-	12.8%	45
Some college	100.0%	77	100.0%	406	27.9%	98
Associate’s degree	-	-	-	-	14.0%	49
Bachelor’s degree	-	-	-	-	35.0%	123
Master’s or higher degree	-	-	-	-	10.3%	36

^*In Experiment 1, Asian and Native Hawaiian categories were inadvertently listed together, the highest age category available was ≥21 years old, and all participants were currently enrolled in college.

^†In Experiment 2, the highest age category available was ≥21 years old, and all participants were currently enrolled in college.

Primary outcome (Knowledge)

In all experiments, and as hypothesized, infographic letter participants were more likely than control letter participants to correctly answer how dropouts affect conclusions, with consistently large absolute percentage differences of ≥15% across experiments (descriptive data, p values, and effect sizes in Table 2; data and confidence intervals in Supplemental Table 2). In Experiment 1, the absolute percentage difference was not statistically significant given the smaller sample but was of similar magnitude.

Table 2.

Sample sizes, primary outcome, secondary outcomes, and discriminant items by letter condition for Experiments 1–3

	Experiment 1 Liberal arts university				Experiment 2 Community college				Experiment 3 Mechanical Turk
	Info- graphic letter	Control letter			Info- graphic letter	Control letter			Info- graphic letter	Control letter
Variables	Estimate % or M ± SD	Estimate % or M ± SD	Absolute difference*	p value Omnibus statistic Cohen’s d**	Estimate % or M ± SD	Estimate % or M ± SD	Absolute difference	p value Omnibus statistic Cohen’s d	Estimate % or M ± SD	Estimate % or M ± SD	Absolute difference	p value Omnibus statistic Cohen’s d
Sample size	42	35			204	202			177	174
Primary outcome
Knowledge, correctly answered question, %	90.5%	74.3%	16.2%	p = .059	63.2%	48.5%	14.7%	p = .003	88.7%	66.7%	22.0%	p < .0001
				X2 = 3.6				X2 = 8.9				X2 = 24.6
				NA				NA				NA
Secondary outcomes†
Transparency of organization, mean	3.8 ± 1.0	2.9 ± 1.1	0.8	p = .0005	3.7 ± 1.1	3.2 ± 1.1	0.6	p < .0001	4.6 ± 0.7	3.6 ± 1.1	0.9	p < .0001
				t = 3.7				t = 5.3				t = 9.7
				d = 0.8				d = 0.5				d = 1.0
Value of retention, mean††	3.8 ± 1.0	3.3 ± 1.0	0.5	p = .018	3.9 ± 1.0	3.7 ± 1.1	0.2	p = .051	4.7 ± 0.6	4.3 ± 0.9	0.4	p < .0001
				t = 2.4				t = 2.0				t = 5.1
				d = 0.5				d = 0.2				d = 0.5
Trust of organization, mean	3.2 ± 0.9	2.3 ± 0.8	0.9	p < .0001	2.9 ± 1.0	2.7 ± 1.1	0.2	p = .049	4.1 ± 0.8	3.7 ± 0.9	0.4	p < .0001
				t = 4.6				t = 2.0				t = 4.8
				d = 1.0				d = 0.2				d = 0.4
Discriminant items†
Value of study outcome, mean†††	4.2 ± 0.7	4.1 ± 0.8	0.2	p = .356	4.2 ± 0.9	4.2 ± 1.0	0.0	p = .970	4.2 ± 0.9	4.2 ± 0.8	0.0	p = .881
				t = 0.9				t = 0.0				t = −0.2
				d = 0.3				d = 0.0				d = 0.0
Keep commitments, mean	3.9 ± 0.8	4.1 ± 0.7	−0.2	p = .395	4.2 ± 0.9	4.2 ± 0.8	−0.1	p = .551	4.5 ± 0.7	4.3 ± 0.8	0.1	p = .108
				t = −0.9				t = −0.6				t = 1.6
				d = −0.3				d = −0.1				d = 0.1

^*Absolute differences in percentages (or in means) between the two letter conditions. Bolded values are statistically significant.

^**Cohen’s d effect sizes were almost identical to the absolute difference in means for each of the continuous secondary outcomes and discriminant items across all three samples given the pooled standard deviations of the differences ranged from 0.7–1.1.

^†Secondary outcomes and discriminant items were assessed with 5-point scales scored 1–5 with the following response options (Not at all, A little bit, Moderately, Quite a bit, Extremely).

^††Value of retention: It is important to me that I complete the survey at the end of the program, whether or not I eat more fruits & vegetables.

^†††Value of study outcome: It is important to me that I eat more fruits & vegetables.

Secondary outcomes

In all experiments, and as hypothesized, infographic letter participants rated the perceived transparency of the research organization higher than controls, with moderate to large effect sizes between conditions across experiments (ds = 0.5–1.0).

In Experiments 1 and 3, and as hypothesized, infographic letter participants rated their perceived value for retention and trust of the research organization higher than control letter participants, with moderate to large effect sizes between conditions across experiments (ds = 0.4–1.0). In Experiment 2, community college students who read the infographic letter did not rate perceived value of retention or trust higher than control letter participants.

Discriminant items

In all experiments, and as hypothesized, infographic letter participants did not value the study outcome more or report keeping commitments more than control participants, with expected minimal effect sizes between conditions (ds = −0.3–0.3).

Conclusions

This study makes two immediate, pragmatic contributions. First, the 1-page infographic—comprised of easy-to-understand, visually powerful graphic images without scientific jargon—could be readily adapted and further tested across a rich array of behavioral medicine outcomes, study designs and delivery channels. Here, knowledge gains were consistently large across three ethnically diverse samples as ~20% more participants correctly answered after only brief exposure to the online infographic letter. Indeed, as many as 91% of infographic participants correctly answered whereas as few as 49% of controls did. Yet, the low initial knowledge for controls also raises important concerns about the current state of participant informed consent and research literacy as trials typically focus on whether research staff follow consent protocols rather than if participants fully understand the rationale for study procedures.

Second, items assessing transparency, perceived value of retention, and trust were sensitive to experimental condition in at least two of three samples for most secondary outcomes, and thus, these items could advantageously be used to assess the role of these important constructs in future large clinical trials, independently of testing infographic strategies per se. Here, transparently explaining the methodological rationale for retention gave rise to greater perceived trust of the research organization, even though the word trust was (purposely) not used in the infographic letter. Given historical abuses by researchers, the role of trust is viscerally important, consistent with long-standing community-based participatory research principles (Israel, Schulz, Parker, & Becker, 1998) and recent qualitative and survey research revealing participants want to be valued partners throughout the research process (Kost et al., 2013).

Several study limitations exist. First, the infographic effect was more consistent across outcomes for Mechanical Turk workers than student samples. This effect could be due to better attention or greater investment (workers are only paid for high-quality work whereas students receive research credit for any participation) or due to different sample demographics such as age or education which interestingly could inform ‘for whom’ infographics may be most effective. Second, this study relied on online convenience samples, 1-time hypothetical scenarios, and self-reported data that may have high response variability, thus limiting if infographic results can be generalized to clinical trial participants with different demographic or health characteristics.

Study strengths included the diverse samples, experimental design, and assessment of discriminant items. Excitingly, future trials could easily embed and then experimentally test whether methodological infographics like the one here improve not only research literacy but actual clinical trial retention rates (Goldberg & Kiernan, 2005), especially among populations with less initial trust about research. Instead, prior research on retention strategies has primarily examined extrinsic motivators such as financial incentives or reminders (Brueton et al., 2014).

In summary, this low-cost, easy-to-understand, and visually powerful infographic appreciably enhanced knowledge, transparency, perceived value of retention, and trust, and could be readily adapted to improve research literacy about retention in accessible online formats in a wide array of clinical trials across behavioral medicine and related disciplines.