Effect of A Mobile Web App on Kidney Transplant Candidates’ Knowledge About Increased Risk Donor Kidneys: A Randomized Controlled Trial

Abstract

Background

Kidney transplant candidates (KTCs) must provide informed consent to accept kidneys from increased risk donors (IRD), but poorly understand them. We conducted a multisite, randomized controlled trial to evaluate the efficacy of a mobile web application, Inform Me, for increasing knowledge about IRDs.

Methods

KTCs undergoing transplant evaluation at 2 transplant centers were randomized to use Inform Me after routine transplant education (intervention) or routine transplant education alone (control). Computer adaptive learning method reinforced learning by embedding educational material, and initial (Test 1) and additional test questions (Test 2) into each chapter. Knowledge (primary outcome) was assessed in person after education (Tests 1 and 2), and 1-week later by telephone (Test 3). Controls did not receive Test 2. Willingness to accept an IRD kidney (secondary outcome) was assessed after Tests 1 and 3. Linear regression Test 1 knowledge scores were used to test the significance of Inform Me exposure after controlling for covariates. Multiple imputation was used for intention-to-treat analysis.

Results

Two hundred-eighty-eight KTCs participated. Intervention participants had higher Test 1 knowledge scores (mean difference=6.61; 95% CI, 5.37–7.86) than control participants, representing a 44% higher score than control participants’ scores. Intervention participants’ knowledge scores increased with educational reinforcement (Test 2) compared to control arm Test 1 scores (mean difference=9.50; 95% CI, 8.27–10.73). After 1 week, intervention participants’ knowledge remained greater than controls’ (mean difference=3.63; 95% CI, 2.49–4.78) (Test 3). Willingness to accept an IRD kidney did not differ between study arms at Tests 1 and 3.

Conclusion

Inform Me use was associated with greater KTC knowledge about IRD kidneys above routine transplant education alone.

INTRODUCTION

As the organ shortage grows, transplant centers are progressively using organs from increased risk donors (IRD).^1,2 IRDs have behavioral risk “factors associated with increased risk for disease transmission, including blood borne pathogens HIV, Hepatitis B (HBV), and Hepatitis C (HCV)” based on the US Public Health Service (PHS) guidelines.³ Approximately 20% of US donor organs are from IRDs.^1,2,4,5

Given the additional, albeit low, risk of disease transmission that IRD organs confer,^6,7 Organ Procurement and Transplant Network (OPTN) policy mandates that transplant clinicians obtain specific informed consent from transplant candidates offered IRD organs. The OPTN has not promulgated guidelines for educational content of this informed consent process, and information disclosed about IRD organs varies across centers.^4-7 Most kidney transplant candidates (KTCs) lack sufficient knowledge about IRD organs and their risks to make informed treatment decisions about IRD organs.^8-11

Increasing KTCs’ knowledge about IRD organs is important clinically and ethically. Clinically, IRD kidneys present real, albeit low, risks of disease transmission. Providing comprehensible information about the risks of IRD kidneys is essential for KTCs to make informed treatment decisions about whether to accept IRD kidney offers. Having the prerequisite knowledge and understanding about IRD organs could facilitate KTCs’ decision-making because knowledge is a precursor to behavior.¹² Accordingly, greater knowledge of IRD kidneys may help KTCs receive a transplant sooner.

Decision aids can effectively increase patients’ knowledge of treatment options, improve comprehension of risks, and generate realistic expectations.^13,14 Two decision aids developed to facilitate transplant candidates’ understanding of IRDs provide personalized risk information about potential wait times¹⁵ or estimated survival by accepting or declining an IRD organ.¹⁶ However, neither aid covers all elements of informed consent, both apply only to the time of organ offer, and neither was evaluated through a randomized controlled trial (RCT).

Mobile health technology “mhealth” (e.g., apps, cellular phones, tablet computers)¹⁷ may overcome such barriers. Mobile apps can increase patients’ health knowledge and behavior through personalized health guidance in other clinical contexts.^18,19 Systematic reviews demonstrate that mHealth interventions effectively increase patients’ knowledge to promote informed consent for routine medical and surgical procedures.^20-22

This multisite RCT aimed to evaluate the efficacy of an iPad application (app), Inform Me, on KTCs’ knowledge about IRD kidneys (primary outcome). We hypothesized that by leveraging mheath, KTCs’ using Inform Me would have knowledge about IRD kidneys comparable to or greater than transplant education alone. We also hypothesized that greater knowledge would be associated with willingness to accept IRD kidneys (secondary outcome). As a neutral decision aid, Inform Me was designed to improve knowledge about IRD kidney risks and foster decision-making in the specific informed consent process, which ethically entails respecting patients’ autonomy to accept or decline IRD kidneys, without undue influence on treatment choice.

METHODS

Study Design

The RCT employed a posttest-only control group design.²³ This design was used because of limited time for data collection in the tight clinic schedule. Inform Me’s efficacy was assessed by comparing the intervention arm with the control arm. On the day of evaluation, intervention arm participants used Inform Me after receiving routine transplant education about IRD kidneys, while control arm participants received routine transplant education about IRD kidneys. Participants received the assigned intervention after completing routine education and clinician visits. All other measures were collected after completing Inform Me (intervention arm) or the knowledge survey (control arm).

Intervention and Setting

The iPad app, Inform Me: about Increased Risk Donor Kidneys (http://mohrlab.northwestern.edu/informme/app/build/), was developed through collaboration between Northwestern University (NU) research investigators and the Center for Behavioral Intervention Technologies at NU. Inform Me used Computer Adaptive Learning (CAL) method to personalize educational materials and content according to each KTC’s comprehension levels in 5 interactive chapters: Introduction, Definition of Increased Risk, Risks and Benefits, Screening for Infection, and Treatment & Follow-Up.^24,25 Chapter content was guided by clinical expertise and prior research.⁸ The Introduction provides an orientation and instructions; the other 4 chapters educate and assess comprehension. Learning objectives guiding educational content were developed for each chapter. Chapter pages included brief textual explanations with interactive tool-tips for additional explanation on the left side of the screen, and videos, interactive graphics, and photographs on the right side for illustration and further explanation. Published data on IRD kidney disease transmission risks, benefits, and tests were transformed into low literacy and low numeracy messages using health communication best practices.^26,27 We focused on IRD kidneys given the larger kidney patient population.

CAL method was used to personalize educational information and knowledge assessment according to each KTC’s comprehension level in interactive sequential chapters (See SDC, File 1 for a description of the Knowledge Test Development).^28,29 The end of each chapter delivers questions to test KTCs’ knowledge of that chapter’s content (Test 1). If, for example, KTCs answered Chapter 1’s Test 1 questions incorrectly, then Inform Me presented additional educational content, specific to topics covered in questions missed, to reinforce learning through repetition. Thereafter, Inform Me presented additional knowledge test questions, corresponding with initially missed question topics, to assess whether the educational reinforcement worked (Test 2) before enabling KTCs to proceed to Chapter 2. CAL method aimed to improve understanding of IRDs by: layering the depth of content to accommodate users with varying functional health literacy, media literacy,³⁰ and information needs; and tailoring education to user’s demonstrated knowledge deficits. Responses to Test 1 and Test 2 were automatically electronically transmitted to an online database for analysis.

Upon completing Test 2, a summary report was generated that presented scores for each chapter, a list of questions answered incorrectly, and themes KTCs should review based on questions missed. Scores were based on Test 1 performance, unless Test 2 was answered; scores derived from the number of items answered correctly. Users may email themselves the summary report along with all questions with correct answers, to reinforce learning.

Northwestern Memorial Hospital (NMH) and University of Alabama, Birmingham (UAB)

NMH’s Kovler Organ Transplantation Center in Chicago, IL, and UAB’s Transplant Center in Birmingham, AL, are large volume kidney transplant programs (performing ~200 kidney transplants per year). Although no gold standard of transplant education exists, both sites engage in similar educational practices and deliver similar content about IRD kidneys, as described in SDC, File 2. All participants received the same routine education. In 2013, during the study enrollment period, 13.6% of all kidney donors were IRDs nationally,¹ while NMH and UAB had 7.5% and 10% IRD kidney utilization rates, respectively.

Recruitment and Participation

Patients consecutively scheduled to attend NMH’s or UAB’s routine education as the first phase of transplant evaluation or re-evaluation were recruited at NMH from October 2013-December 2014, and at UAB from January 2014-July 2014. New patients scheduling an appointment were sent a flyer describing the study in their welcome package by clinic staff. In reminder calls, clinic staff informed patients about the study and asked if a researcher could call them about it. If patients agreed, clinic staff notified research staff, who mailed patients a letter inviting study participation, followed by a phone call.

Re-evaluation patients (candidates who returned for reassessment after being waitlisted several years) were eligible and participated at NMH, but were not recruited at UAB due to limited resources. Re-evaluation patients were mailed a recruitment letter when scheduled in advance, or were recruited and screened in person. Recruitment letters described the study purpose and what participation entailed, and provided a telephone number for individuals to ask questions or to opt out of the study. Research staff called all patients to screen for eligibility and assess interest in participation.

Individuals were eligible if they were: age 21 years and older, English speaking, never received a kidney from an IRD, reported ‘never’, ‘rarely’, or ‘sometimes’ to the health literacy question: “How often do you need to have someone help you when you read instructions, pamphlets, or other written material from your doctor or pharmacy?"³¹ and willingness to use an iPad 2 tablet. Some reading ability was essential for participants to effectively use the iPad.

As KTCs arrived at the transplant center, research staff confirmed their eligibility and obtained written informed consent. Staff then randomized participants, using 1:1 equal allocation, to receive either routine education only (control arm) or Inform Me after attending routine education (intervention arm), using a computer-generated random number list, with individual numbers inserted into sequentially numbered, sealed envelopes concealed until study arm was assigned. Randomization was stratified by site.

Next, all participants completed a self-administered 25-item paper survey assessing sociodemographics (gender, age, highest level of education completed, self-identified race, ethnicity, total household income, prior consideration of living kidney donation, marital status, employment status, and primary insurance), health literacy, and Internet self-efficacy. Internet self-efficacy served to control for users’ ability to use the app; those more adept may be better able to gain knowledge than those less adept.

The trial was single blinded; research team members assessing outcomes (EJG, MWS, MGI) were blinded to assignments to the intervention. For our power analysis, our sample size estimate was to recruit 100 participants in each arm to have 80% power to detect 30% difference in knowledge scores, assuming a coefficient of variation = 0.5 for a 2-tailed test Type 1 error rate of 0.05 in posttest-only data. Although 1:1 randomization was implemented, we realized that, due to technical problems with Internet connectivity and concomitant concerns over potential data loss for the intervention arm, we needed to recruit more participants to ensure at least 100 participants per arm. We therefore generated an additional 100 random numbers, which were mostly used at the NMH site. The study stopped recruitment after reaching our initial target sample size, twelve numbers were not used (6 per site). We recovered most data and obtained a larger sample than our initial recruitment target. An interim analysis was conducted for presentation at conferences.^32,33 The Institutional Review Board at NU and at UAB approved this study.

Test Protocol

After routine transplant education, control arm participants completed the 31-item multiple choice posttest (Test 1) on paper, while the intervention arm completed the Inform Me app, which included the identical Test 1 in electronic format. Only the intervention arm completed additional questions (Test 2), in electronic format. Immediately thereafter, participants completed a self-administered paper survey on willingness to accept an IRD kidney offer. Intervention arm participants were also asked 3 open-ended questions about their experience, and what they liked and disliked about using Inform Me as a process evaluation of the intervention. Such feedback is important for improving the app and/or its delivery in future research studies. Data collection occurred in the transplant waiting room, a patient clinic room, or the education room. Research staff showed users how to use the app and remained present to answer further questions. Participants were compensated with a $30.00 gift card.

To identify and troubleshoot problems with the intervention delivery, research assistants filled out a form for each intervention arm participant. Data were used solely to foster communication within the research team to maintain fidelity of the protocol. For example, early in the study, 2-3 intervention arm participants accidentally hit the browser home button, causing the Inform Me application to inadvertently close before completion. Research staff notified the research team to obtain IT assistance to disable the home button, thereby preventing future accidents.

All participants completed a telephone follow-up posttest (Test 3) 1 week later that was identical to Test 1, including aforementioned survey measures. Intervention arm participants were also asked the same 3 open-ended questions about using Inform Me and if they had reviewed the Summary Report in the past week. We made at least 5 phone call attempts during the day and evening to reach participants. Participants were compensated with a $35.00 gift card.

Independent Measures

Newest Vital Sign (NVS)³⁴ assessed health literacy/numeracy. NVS asks 6 questions about nutrition label information. NVS has good internal consistency (Cronbach’s α>0.76), criterion validity (r=0.59),³⁴ and high external validity.^34,35 Scores range from 0 - 6.³⁴ Scores are trichotomized into: low literacy (0–1), moderate literacy (2–3), and adequate literacy (4–6).

eHealth Literacy Scale (e-HEALS) assessed participants’ self-efficacy with using the Internet to search and understand health information.^36,37 The validated 8-item, 5-point Likert-type scale has good internal consistency (Cronbach’s α=0.88) and test-retest reliability=0.49.³⁶ Scores range: 1–40. Scores are trichotomized into low (1–22), medium (23–32), and high (32–40) efficacy.

Outcome Measures

Knowledge about IRD kidneys was measured by a 31-item multiple choice test. Scores range: 0–31; higher scores reflect greater knowledge.

Willingness to accept a hypothetical IRD kidney offer was measured by a 5-point Likert scale item anchored by ‘Strongly agree’ and ‘Strongly disagree.’ Scores range: 1–5; lower scores reflect greater willingness.

Statistical Analysis

The primary outcome, mean difference in posttest knowledge scores, was used to compare intervention and control arms. Our analysis was based on the intention-to-treat principle and included all persons who were randomized regardless of whether they completed the posttest assessments. We tested the pattern of missing values in Test 1 knowledge scores using a multivariable logistic regression to predict the probability of missing values as a function of all covariates in Table 1. None of them were significantly associated with the probability of Test 1 knowledge scores missing. Because they were missing at random, we imputed them using multiple imputation based on a Bayesian iterative Markov Chain Monte Carlo (MCMC) method^38,39 with 20 repetitions. Knowledge scores (Tests 1–3) were separately analyzed using linear regression as a function of study arm assignment and study site (NMH versus UAB). We used the same models to analyze willingness to accept an IRD kidney (Test 1 and Test 3).

Table 1.

Participant Demographic Profile by Study Arm and Study Site

Demographic Characteristics	All		Control		Intervention		P-Value
N (%)*	288	(100.0)	155	(53.8)	133	(46.2)
Age, mean/median (sd)	50.8/52	(11.8)	50.5/51	(12.3)	51.2/53	(11.3)	0.61
Male, n (%)	176	(61.1)	97	(62.6)	78	(58.6)	0.61
Race/ethnicity, n (%)
NonHispanic White	93	(32 .3)	51	(32.9)	42	(31.6)	0.94
NonHispanic Black	158	(54.9)	85	(54.8)	73	(54.9)
Other	37	(12.8)	19	(12.3)	18	(13.5)
Marital Status, n (%)
Married/Living with SO	129	(44 .8)	75	(48.4)	54	(40.6)	0.19
Not Married	159	(55.2)	80	(51.6)	75	(56.4)
Education, n (%)
Less than college	208	(72 .2)	111	(71.6)	97	(72.9)	0.80
BA or higher	80	(27.8)	44	(28.4)	36	(27.1)
Household Income, n (%)
< $25,000	146	(50 .7)	75	(48.4)	71	(53.4)	0.11
$25,000 - $64,999	63	(21.9)	42	(27.1)	21	(15.8)
>= $65,000	57	(19.8)	26	(16.8)	31	(23.3)
Unknown	22	(7.6)	12	(7.7)	10	(7.5)
Health Insurance, n (%)
Private Insurance/Group Policy	88	(30 .6)	49	(31.6)	39	(29.3)	0.67
Public Insurance, Other	200	(69.4)	106	(68.4)	94	(70.7)
Current Work Status, n (%)
Working	82	(28 .5)	37	(23.9)	45	(33.8)	0.06
Not Working	206	(71.5)	118	(76.1)	88	(66.2)
Health Literacy
Adequate	155	(53 .8)	87	(56.1)	66	(49.6)	0.32
Moderate	73	(25.3)
Inadequate	60	(20.8)	41	(22.0)	19	(18.6)
e-Heals, n (%)
Low	42	(14.6)	18	(11.6)	24	(18.0)	0.19
Medium	146	(50.7)	76	(49.0)	70	(52.6)
High	68	(23.6)	43	(27.7)	25	(18.8)
Unknown	32	(11.1)	18	(11.6)	14	(10.5)
Re-Evaluation Patient
Yes	17	(5.9)	6	(35.3)	11	(64.7)	0.35
No	271	(94.1)	127	(46.9)	144	(53.1)

^*Row percent; all other percent values are column percent

^**P-Values were obtained from chi-square test for categorical variables and t-test for continuous variables (age, health literacy)

Our study arms were well-balanced in terms of participant demographic characteristics, health literacy, and Internet self-efficacy. In all regression analyses, we adjusted for all of these characteristics. For sensitivity analysis, we estimated all models using only the complete cases after dropping all participants with any missing values. SDC, Table 1 presents frequencies of missing data. All statistical analyses were conducted using Stata SE version 14 (StataCorp, College Station, TX). Statistical significance was established at a 2-tailed alpha of .05.

Qualitative Analysis

The brief responses to open-ended items in surveys were coded by hand by an expert in qualitative research (EJG) twice (separated by 1 month) to check consistency in the identification of themes inductively emerging from the data⁴⁰ until reaching saturation.

RESULTS

Participant Demographic Characteristics

Across both sites, 468 (286 at NMH, 182 UAB) of 593 KTCs invited to participate were eligible (Figure 1). KTCs were ineligible primarily because they had vision problems, were not comfortable using an iPad, or did not speak English. Of all eligible KTCs, a total of 288 patients participated (62% overall; 194/286 (68%) at NMH, 94/182 (52%) at UAB). Participants were randomized into the intervention arm (n=133) or the control arm (n=155). One person did not receive the Inform Me intervention, and 17 people did not complete the Inform Me intervention.

Figure 1. Consort Diagram.

Most participants were male (61.1%), African American (54.9%), and the mean age was 51 (range: 22–77) (Table 1). Seventeen participants were re-evaluation patients. While most participants had adequate literacy (53.8%), about a quarter had high (23.6%) Internet self-efficacy. Among our study participants (n=280), internal consistency was good for health literacy (Cronbach’s α>0.75), and Internet self-efficacy (Cronbach’s α>0.95). Significant demographic differences did not arise between control and intervention arms, but did arise between study sites in most characteristics (Table 1). Intervention arm noncompleters were similar demographically to completers (data not shown). Demographic differences between participants and nonparticipants could not be assessed as nonparticipants did not provide demographic information.

Initial Test Scores After Randomization (Test 1)

The mean total knowledge score was 17.06 (range: 1–30) following either routine education only (control) and/or completing initial questions in Inform Me (intervention). The mean ± SD, interquartile range (25%–75%), and minimum-maximum range to complete Inform Me was 47 ± 21, (Median=42, IQR=30–60), (min-max range: 16–100) minutes. Knowledge scores were significantly greater among whites (p<0.001), and participants who had some college or higher education (p<0.001), higher incomes (p<0.001), adequate health literacy (p<0.001), higher Internet self-efficacy (p<0.02), and other characteristics (Table 2). No significant differences in knowledge scores arose between study sites or between new versus re-evaluation patients. Intervention arm participants had a mean 6.61 points (95% CI, 5.37–7.86, p<0.001) higher initial knowledge score (Test 1) compared to control arm participants after controlling for all covariates (Table 3). This represents a 44% greater knowledge score associated with Inform Me exposure (effect size d=1.15). Knowledge scores for individual chapters were significantly greater in the intervention arm (SDC Table 2).

Table 2.

Mean (Standard Error) of Knowledge Score (Test 1)

Demographic Characteristics	All		P-Value*	Study Arm
				Control		Intervention		P-Value
N (%)	288	(100.0)		155	(53.8)	133	(46.2)
Overall	17.06	(0.40)		13.94	(0.42)	20.69	(0.57)	< 0.001
Age Categories
≤ 40	18.58	(0.84)	0.121	15.49	(0.92)	22.49	(1.07)	< 0.001
41 - 50	17.40	(0.80)		14.42	(0.81)	22.02	(1.15)	< 0.001
51 - 60	16.71	(0.67)		13.58	(0.75)	19.41	(0.91)	< 0.001
61 or older	15.84	(0.85)		12.44	(0.78)	20.04	(1.30)	< 0.001
Sex
Female	16.95	(0.61)	0.84	14.10	(0.69)	20.07	(0.85)	< 0.001
Male	17.12	(0.52)		13.85	(0.52)	21.08	(0.73)	< 0.001
Race/ethnicity
NonHispanic White	19.26	(0.72)	< 0.001	15.88	(0.81)	23.36	(0.94)	< 0.001
NonHispanic Black	16.12	(0.50)		13.15	(0.47)	19.57	(0.74)	< 0.001
Other	15.54	(1.14)		12.30	(1.33)	18.97	(1.46)	0.002
Marital Status
Not Married	16.95	(0.53)	0.76	13.85	(0.55)	20.09	(0.75)	< 0.001
Married/Living with Significant other	17.19	(0.60)		14.04	(0.63)	21.56	(0.82)	< 0.001
Education
High school or less	16.03	(0.45)	< 0.001	12.65	(0.43)	19.90	(0.63)	< 0.001
Some college or higher	19.73	(0.74)		17.20	(0.79)	22.81	(1.08)	< 0.001
Household Income
< $25,000	15.60	(0.52)	< 0.001	12.78	(0.52)	18.59	(0.77)	< 0.001
$25,000 - $64,999	16.35	(0.72)		13.73	(0.64)	21.58	(1.00)	< 0.001
>= $65,000	21.45	(0.83)		17.67	(1.17)	24.62	(0.83)	< 0.001
Unknown	17.35	(1.82)		13.86	(1.99)	21.54	(2.37)	0.02
Health Insurance
Private Insurance/Group Policy	19.38	(0.70)	< 0.001	15.87	(0.77)	23.79	(0.83)	< 0.001
Public Insurance, Other	16.03	(0.46)		13.05	(0.48)	19.40	(0.66)	< 0.001
Current Work Status
Not Working	15.83	(0.45)	< 0.001	13.30	(0.45)	19.23	(0.71)	< 0.001
Working	20.13	(0.71)		15.99	(0.92)	23.54	(0.75)	< 0.001
Health Literacy
Adequate	18.36	(0.52)	< 0.001	15.36	(0.55)	22.30	(0.73)	< 0.001
Moderate	16.14	(0.74)		12.04	(0.73)	19.70	(0.88)	< 0.001
Inadequate	14.71	(0.84)		12.21	(0.85)	17.76	(1.33)	< 0.001
Internet Self-Efficacy
Low	16.27	(0.97)	< 0.02	13.13	(1.38)	18.63	(1.17)	0.005
Medium	17.68	(0.56)		13.91	(0.53)	21.76	(0.77)	< 0.001
High	17.73	(0.77)		15.36	(0.84)	21.80	(1.19)	< 0.001
Unknown	13.84	(1.08)		11.49	(1.23)	16.85	(1.54)	0.01

Table 3.

Comparison of Outcome Measures by Study Arm

Variable	Bivariate, Mean (95% CI)						Adjusted for all Covariates*
	All Participants		Control		Intervention		Mean Difference (95% CI)		P-Value
Inform Me, Test 1	17.04	(16.27 - 17.84)	13.94	(13.12 - 14.76)	20.69	(19.56 - 21.81)	6.61	(5.37 - 7.86)	< 0.001
Inform Me, Test 2	18.33	(17.50 - 19.16)	13.94	(13.12 - 14.76)	23.44	(22.51 - 24.38)	9.46	(8.32 - 10.60)	< 0.001
Inform Me, Test 3	16.20	(15.52 - 16.87)	14.70	(13.90 - 15.51)	17.94	(16.90 - 18.99)	3.63	(2.49 - 4.78)	< 0.001
Willingness to accept an IRD kidney, Test 1	2.68	(2.54 - 2.83)	2.78	(2.58 - 2.97)	2.57	(2.34 - 2.81)	−0.20	−(0.53 - 0.12)	0.22
Willingness to accept an IRD kidney, Test 3	2.69	(2.53 - 2.85)	2.81	(2.61 - 3.01)	2.54	(2.29 - 2.80)	−0.28	−(0.61 - 0.04)	0.09

^*Adjusted for study site, age, sex, race/ethnicity, marital status, education, household income, health insurance, health literacy, e-Heals.

P-values < 0.05 are in bold. All p-values < 0.01 would indicate significant difference after Bonferroni correction for multiple comparison.

Although the control arm did not take Test 2, Test 1 data are also presented in Test 2 for the control arm in the table to facilitate comparison between the intervention arm at Test 2 and control arm at Test 1.

While intervention arm participants indicated greater willingness to hypothetically accept an IRD kidney than control arm participants, the relationship was insignificant (2.57 versus 2.78, p=0.22).

Reinforcement Questions from Inform Me (Intervention Arm Only) (Test 2)

After additional education, intervention arm knowledge scores increased significantly from Test 1 to Test 2 by 2.75 points (p=0.001), and were greater than the control arm Test 1 scores by a mean of 9.50 (8.27–10.73) (p<0.001) points.

Follow-Up Testing 1 Week after Randomization (Test 3)

One week after routine education, intervention arm participants’ knowledge remained a mean of 3.63 (2.49–4.78) points greater than the control arm (p<0.001). Knowledge scores for most chapters remained significantly greater in the intervention arm (SDC Table 2).

Although intervention arm participants had greater willingness to accept an IRD kidney, the difference approached, but did not reach, significance (2.54 versus 2.81, p<0.09).

Covariates

The regression model of Test 1 scores was adjusted for age, race/ethnicity, education, income, employment, health literacy, Internet efficacy, study site, insurance, gender, and marital status (Table 4). Participants exposed to Inform Me had a 6.6-point greater knowledge score than those in the control arm (p<0.001).

Table 4.

Multiple Linear Regression Analysis of Test 1 Knowledge S core (N = 288)^*

Variable	Coefficient (95% CI)	P-Value
Intervention	6.612 (5.366 - 7.859)	< 0.001
Study Site UAB [NMH]	0.962 −(0.367 - 2.292)	0.16
Age [< 40]
40 - 49	−1.458 −(3.204 - 0.289)	0.10
50 - 59	−2.382 −(4.040 - -0.725)	0.005
60 or older	−2.616 −(4.557 - -0.675)	0.009
Male [Female]	0.230 −(1.004 - 1.465)	0.71
Race/ethnicity [NonHispanic White]
NonHispanic Black	−0.959 −(2.561 - 0.642)	0.24
Other	−2.520 −(4.536 - -0.503)	0.02
Married or Living with Significant Other [Living Alone]	−0.359 −(1.633 - 0.915)	0.58
Some college or higher [High school or lower]	2.392 (0.893 - 3.892)	0.002
Income [< $25,000]
$25,000 - $64,999	0.604 −(1.111 - 2.318)	0.49
>= $65,000	2.743 (0.693 - 4.794)	0.009
Unknown	2.047 −(1.015 - 5.109)	0.17
Public or Other [Private]	−0.789 −(2.363 - 0.784)	0.32
Employed [Unemployed]	1.474 −(0.012 - 2.960)	0.05
Health Literacy [Adequate]
Moderate	−0.856 −(2.343 - 0.631)	0.257
Inadequate	−1.639 −(3.363 - 0.085)	0.06
Internet efficacy [Low]
Medium	0.046 −(1.664 - 1.756)	0.96
High	−0.617 −(2.685 - 1.452)	0.56
Unknown	−2.633 −(5.068 - -0.198)	0.03

^*Reference categories are inside angle brackets.

Participants less than age 40 had a 2-point greater knowledge score than those age 50–59 (p=0.005), and an almost 3-point greater knowledge score than those age 60 and older (p=0.009). Non-Hispanic Whites had a 2-point greater knowledge score than other, non-African American racial/ethnic groups (p=0.02). Participants with some college had a 2-point greater knowledge score than those with a high school degree or less (p=0.002). KTCs with incomes over $65,000 had an almost 3-point greater knowledge score than those with incomes less than $25,000 (p=0.009). Employed participants had a 1-point greater knowledge score than those unemployed (p<0.05). Knowledge scores did not differ significantly between study sites.

Sensitivity Analyses

For the sensitivity analysis, we estimated a multiple regression model with all covariates using only complete cases and compared the results with those based on multiple imputation (SDC, Table 3). The results were very similar. The mean difference in knowledge score in the complete case analysis was 6.27 points (95% CI, 4.92–7.62; p<0.001) compared to 6.61 points (95% CI, 5.37–7.86; p<0.001) in the multiple imputation analysis.

We tracked the time intervention arm participants required to complete the app to assess whether time was associated with improved knowledge. Duration of app use time was not correlated with knowledge (r=−0.03, p=0.79 in Test 1; r=−0.02, p=0.84 in Test 2), or predictive of knowledge scores in multivariable regressions.

Qualitative Themes Regarding KTCs’ Experiences Using Inform Me

Regarding intervention arm participants’ comments about experiences using Inform Me, participants reported mostly positive (85%), and few negative (11%), or miscellaneous (4%) comments. Regarding what participants liked about Inform Me, comments were mostly positive (89%), and few were negative (2%), or miscellaneous (9%). Inform Me was informative, well-liked, easy to use and understand, and presented in an enjoyable multimedia format. Regarding what participants disliked about Inform Me, comments were mostly negative (97%), or miscellaneous (3%). Respondents did not like Inform Mes long length and reported that videos were slow or did not work given transplant center Internet connectivity problems.

DISCUSSION

This multi-site study demonstrated that a mobile web app, Inform Me, resulted in 44% greater knowledge with an effect size of 1.15, which is large for behavioral health interventions.^41,42 Inform Me is a clinically relevant decision aid that delivers information necessary for KTCs to provide OPTN-mandated specific informed consent. By improving KTCs’ comprehension of the risks, benefits, and alternatives to accepting or refusing IRD kidneys,^8,9 Inform Me can promote patient autonomy and patient-centered care.^43,44 Inform Me appeared well-liked overall.

Systematic reviews show that web-based and other interactive interventions were significantly more effective than routine education in increasing knowledge as a prerequisite to informed consent.^20-22 We believe that the app, rather than repeated clinician education or paper-based educational materials, was critical to generating our high effect size. Unlike static educational materials, Inform Me uses CAL method to tailor and reinforce educational content and knowledge test items to patients own information needs. Unlike paper, Inform Me shows videos, animations, and graphics to depict complex concepts, which accords with recommended multisensory patient education approaches.^45,46

Inform Me standardizes education about IRD kidneys across transplant centers, a benefit of electronic-based interventions.²⁰ Transplant centers vary in their education and informed consent processes,^47,48 which can generate suboptimal communication. As a standardized tool, Inform Me can ensure that all patients are exposed to the same critical information.

Transplant centers could use Inform Me to supplement current education practices to enhance patient comprehension about IRD kidneys. Inform Me could be used in diverse clinical settings: as a clinic-based program, provided in tablets, at a computer kiosk in the waiting room, or made freely available on a website or smartphone for use at the patient’s convenience. Its use in such settings should be assessed, and some patients may not have access to the Internet.

Participants’ knowledge retention dropped 1 week postintervention, consistent with other studies.²² Although the intervention arm’s knowledge score remained significantly higher than the control arm’s at 1-week, the knowledge drop raises concern as to whether Inform Me can affect future decision making and acceptance of IRD kidneys. Therefore, using Inform Me multiple times as a ‘booster shot’ may reinforce learning (e.g., at initial evaluation, re-evaluation, and near the organ offer to remind patients of their options and have patients reiterate their decisions), should also be evaluated. Studies show that greater access to transplant education resources and greater transplant knowledge can foster patients’ evaluation completion.^49,50 Providing Inform Me after patients initiate evaluation and attain basic transplant knowledge may help them stay focused and not feel overwhelmed. Inform Me summary reports may be reviewed to facilitate communication between patients and providers, and included in medical records to document informed consent.

Moreover, we found that Inform Me can be used by diverse patient populations across all health literacy levels. However, adequate health literacy was independently related to greater knowledge scores, suggesting that KTCs with inadequate literacy may need assistance with comprehending written or statistical information. Internet self-efficacy was unrelated to knowledge scores suggesting that Inform Me was navigable by all participants.

It is unclear whether the intervention arm’s greater knowledge scores were a function of greater time in learning, the mode of delivery, or both. Duration of app use time was not significantly correlated with knowledge. Thus, improvement in test performance was not a function of increased learning time, but likely a function of the tailored educational approach. The increase in the intervention arm’s knowledge scores from Test 1 to Test 2 through CAL method suggests that tailoring education to patients’ information needs worked. The average 47-minutes to complete Inform Me fostered patient-centered learning, as patients elected to learn more, but would likely not fit within transplant centers’ tightly coordinated clinical schedules. Ongoing efforts will be made to shorten Inform Me, facilitate implementation into clinical practice, and update content affecting risk-benefit analysis (i.e., broad availability of directly acting antivirals for HCV).

Inform Me was associated with greater, albeit not significantly, willingness to accept an IRD kidney, which increased from Test 1 to Test 3. Our study was not powered to test for willingness; future studies should assess this variable. Non-neutral decision aids¹¹ designed to encourage patients to accept IRD organs¹⁵ can compromise informed consent.⁵¹ Inform Me’s neutral design can allay policy-makers’ concerns that patients are well-informed about IRD organs given the potential for infectious disease transmission.^52,53

Study strengths include the multi-site design, no center-specific effect, and representation of a diverse patient population in 2 geographically different U.S. regions, suggesting broad generalizability of our results. However, there are limitations. It is unclear whether the app itself or additional education above and beyond routine education contributed to knowledge improvement. While clinicians spending more time discussing the content might have accomplished the same goals, we found that the app constitutes an effective approach to educate patients requiring little clinician time and resources. Inform Me does not address concerns about non-renal IRDs. Future research will adapt this app to other organs. Although the posttest-only control group design is not as strong as a pretest-posttest control group design, it remains a true experimental design addressing all sources of internal invalidity,²³ and a pretest before the clinic visit would have been burdensome and not practicable as clinical evaluation lasts all day. We examined immediate outcomes but not proximal outcomes (e.g., decision-making). Inform Me likely provided more in-depth information about IRD kidneys than in routine education, which may have biased the intervention arm’s knowledge scores.

Study participants could have been more computer-savvy than nonparticipants, which may have amplified the intervention’s effect. Seventeen people did not complete the intervention, possibly due to its length. Without having conducted a pretest, it is unclear whether intervention noncompleters differed from completers regarding fears or attitudes about IRD kidneys, even though they did not differ demographically. Our 12.8% attrition rate is higher than or comparable to similar studies.^54-56 A shorter app may be more acceptable for patients. Future research will assess patients’ perceptions of the acceptability of the time duration.

The control arm had more participants than the intervention arm. Twelve numbers were not used in the additional random numbers, which may explain the uneven number of subjects across study arms. The study arms may not be completely equivalent from not using all random numbers, even though they did not differ demographically. Participants may have had prior transplants, exposing them to information about IRD organs and influenced their current knowledge. However, prior transplant education was unlikely to have had an impact because routine education duration about IRD kidneys (NU versus UAB), and intervention duration were both unrelated to the intervention arm’s greater knowledge. KTCs who are positive for Hepatitis B, Hepatitis C, or HIV may have greater knowledge about these diseases and risks of disease transmission. However, we did not assess whether participants were positive for these diseases, which could have influenced knowledge scores. Future research should control for this variable. Multiple coders were not used to confirm themes, which was a limitation. Despite these limitations, this is the first prospective, patient-centered RCT about IRD knowledge.

We did not assess the impact of Inform Me on actual decisions about accepting IRD kidneys, which is not feasible because organs are offered unexpectedly. Future research should assess the impact of Inform Me on KTCs’ decision making (accept/decline IRD kidneys, decisional conflict), outcomes (satisfaction, decisional regret), and patient-clinician communication (number of questions asked, duration) when obtaining patients’ informed consent for IRD kidneys, before evaluating the uptake of this app into clinical practice.²¹ Furthermore, because NU and UAB did not have a policy requiring candidates to provide consent for accepting IRD kidneys at the time of listing, we were unable to assess whether app exposure affected consent rates ahead of the organ offer. Future research should assess the latency of the learning and the optimal frequency that patients should re-access the education offered through processes including the app. As Inform Me was developed in English, future research should translate it into other languages and evaluate performance.

Conclusion

Our mobile web app, Inform Me, was associated with KTCs’ greater knowledge about IRD kidneys in addition to that obtained through routine transplant education. This study demonstrated the feasibility of using Inform Me in the transplant clinic setting. Inform Me holds promise to supplement transplant education.

FOOTNOTES.

• Participated in research conception and/or design: Gordon, Ison, Sohn, Chang

• Participated in the writing of the paper: Gordon, Ison, Mannon, Sohn, Chang

• Participated in the performance of the research, Gordon, Vera, Mannon

• Participated in data analysis and interpretation of data: Gordon, Ison, Sohn, Chang, Vera

• Participated in statistical analysis: Gordon, Sohn

• Reviewed the final manuscript: Gordon, Ison, Chang, Sohn, Beauvais, McNatt, Warren, Mannon, Vera

• Provided administrative, or technical support: Ison, Mannon

Abbreviations

App

Application

HBV

Hepatitis B Virus

HCV

Hepatitis C Virus

HIV

Virus

IRD

Increased Risk Donor

KTCs

Kidney Transplant Candidates

NMH

Northwestern Memorial Hospital

OPTN

Organ Procurement and Transplantation Network

PHS

Public Health Service

UNOS

United Network for Organ Sharing

UAB

University of Alabama, Birmingham