Impact of financial reimbursement on retention rates in military clinical trial research: A natural experiment within a multi-site randomized effectiveness trial with active duty service members

Abstract

Introduction

Achieving adequate retention rates in clinical trials is essential to ensuring meaningful results. Although financial reimbursement is an effective strategy to increase participant retention, current policies restrict the use of federal funds to reimburse U.S. active duty Service members for research participation. It is unknown whether permitting financial reimbursement among this population would improve trial retention rates. A recent randomized effectiveness trial received approval to provide reimbursement to Service member participants several months after recruitment began, creating a natural experiment to study the effects of financial reimbursement on retention.

Materials and methods

Active duty Service members recruited from six U.S. military treatment facilities (N = 666) were enrolled in a collaborative care study and completed assessments at baseline, three-, six-, and 12-months. Data on study assessment completion rates at three- and six-months were analyzed using the mixed-effects binary logit model to determine the probabilities of completing assessments based on reimbursement status.

Results

Participants who received reimbursement were significantly more likely to complete study assessments at both time-points than participants who did not receive reimbursement (p < 0.01). Survey completion was 5% and 4% greater among participants offered reimbursement at three- and six-month time-points, respectively.

Conclusion

Results suggest that providing Service members with reimbursement for research participation is associated with modest increases in retention rates in clinical trials. Findings provide useful insight for researchers, funding agencies, and policy-makers in considering retention strategies to maximize the value and impact of military research.

1. Introduction

Clinical trials are necessary to determine if treatments are safe and efficacious [1]. Meaningful clinical trials are typically resource-intensive, and adequate recruitment and retention rates are required to achieve sufficient power and minimize bias. However, recruitment and retention in clinical research is notoriously difficult. Indeed, a recent review found that only 55% of 73 publicly funded randomized controlled trials (RCTs) in the United Kingdom met proposed recruitment targets [2]. Recruitment rates set a ceiling on participant retention, both of which are important for robust findings.

Conducting clinical research with Service members is essential to understand the unique effects of military service and to provide effective care. Similar to other populations, controlled trials of active duty Service members often report recruitment difficulties [3,4] and high research attrition rates [[5], [6], [7]]. Based on the considerable investment placed in military research and the national value of improving care for Service members, it is important to evaluate strategies that help retain Service member participants to maximize validity and generalizability of results that can ultimately inform evidence-based practice and policy decisions affecting this population.

Empirical evidence suggests that monetary reimbursement is an effective strategy to improve recruitment and retention rates in research trials. A recent Cochrane systematic review evaluating 38 randomized trials and six strategies to improve study retention concluded that monetary incentives were the only effective method [8]. Another systematic review evaluating incentive strategies in longitudinal cohort studies found monetary incentives were associated with a 2%–13% increase in retention rates [9]. Finally, a third systematic review of RCTs designed to increase response to questionnaires found the odds of response to an electronic questionnaire almost doubled when gift cards were offered [10]. Despite reported benefits of monetary incentives to increase trial participation and retention, current compensation policies within the DoD restrict the reimbursement of active duty Service members for participation in research. Per DoD Instruction 3216.02, on-duty Service members may be compensated for blood draws, but not for general research participation via direct federal funding [11]. Payment for research participation is only permissible for off-duty Service members, if provided by a federal contractor or non-federal source [11]. It is unclear how this policy impacts study recruitment and execution, and thus the validity of results. Research evaluating the effect of current DoD policies on research participation reimbursement may help inform future policy decisions surrounding such procedures in military populations.

Recently, a large randomized effectiveness trial on collaborative care in an active duty sample achieved retention rates of 87%–93% over one-year follow-up [12]. Study design characteristics and a mid-trial change in policy of financial reimbursement created a natural experiment opportunity to evaluate the impact of reimbursement on longitudinal survey completion. This paper examines the association between financial reimbursement and assessment completion among participants enrolled in the trial. Investigators hypothesize that individuals who were offered financial reimbursement for completing study assessments would have a greater probability of assessment completion relative to participants who were not offered reimbursement.

2. Methods

2.1. Design

This study represents a secondary data analysis from a large multi-site collaborative care study; details on the trial design and main outcomes are available elsewhere [12,13]. Briefly, participants were 666 active duty Service members who screened positive for post-traumatic stress disorder (PTSD) and/or depression during routine primary care visits across six large U.S. military treatment facilities (18 primary care clinics); participants were randomly assigned to either the enhanced collaborative care or usual care treatment arm [13]. Data were collected at baseline and three-, six-, and 12-months from time of enrollment, and each assessment was estimated to require approximately 1 h for participants to complete [12]. Follow-up assessment windows were open 30 days before and 60 days after the follow-up date to account for the challenging schedules of active duty Service members and the potential that they may be unable to complete study assessments at the exact follow-up time. When participants entered the follow-up window they received e-mail, telephone, and text message reminders, which were alternated and sent at different times of day throughout the follow-up period. Multiple assessment formats were also available to participants, including a secure web portal option, interview with a study coordinator via telephone, or mailed paper-and-pencil questionnaire. Follow-up assessment formats were staggered based on participant response to reminder e-mails, phone calls, and text messages. Participants always had the option to complete study assessments via secure web portal throughout the follow-up period. After five assessment reminders, a research coordinator called participants who had not yet responded to complete their assessment via telephone interview. After 11 reminders, participants were mailed the paper-and-pencil packet to complete and return via pre-paid envelope. The study was approved by Institutional Review Boards (IRBs) at Walter Reed National Military Medical Center (lead), six participating Army installations, RTI International, RAND Corporation, University of Washington, Boston VA, and the Human Research Protection Office, U.S. Army Medical Research and Materiel Command. All participants provided informed consent prior to study participation.

Approximately 10 months after study recruitment started, an amendment was approved by the lead IRB allowing participant reimbursement to offset the burden associated with completing study assessments. This change produced variation in whether participants were reimbursed at each of the four time-points. Reimbursement was provided by a non-federal source in the form of a gift card to an online retailer, and instructions for obtaining the gift card were delivered to participants via e-mail. After participants completed a study assessment during off-duty hours, they received an activation code that allowed them to retrieve and use their online gift card from a website. Reimbursement for completing study assessments was offered in escalated amounts over the course of one year. Participants received a gift card worth $40 for completing the eligibility assessment, a $45 gift card for completing the three-month follow-up, a $50 gift card for completing the six-month follow-up, and a $55 gift card for completing the 12-month follow-up assessment, for a possible total of $190 in online gift cards for study participation over one year. Gift card amounts were determined based on participant time needed to complete assessments, sensitivity of questions asked in the assessments, and available study funding, with caution to avoid potential coercion. Participants already enrolled in the study before reimbursement was approved were retroactively issued online gift cards for the assessments they had already completed.

2.2. Sample

All 666 participants randomized in the trial were included in the analyses. Baseline characteristics of the overall study sample can be found elsewhere [12]. In general, participants were primarily enlisted males, between ages 20 and 30, with at least a high school education – demographics consistent with the general military population [12]. In the present study, participants were defined based on their reimbursement status at three- and six-month follow-up. Prior to IRB approval of reimbursement, 237 participants were eligible to complete the three-month follow-up assessment, and 85 participants were eligible to complete the six-month follow-up assessment. These participants were included in the non-reimbursement group at each time-point and compared to participants who were eligible to receive reimbursement at each follow-up period (n = 429 participants at three-month follow-up; n = 581 participants at six-month follow-up). The 12-month follow-up was not included in the analyses because no participants had reached that window before reimbursement was approved.

2.3. Statistical analyses

Descriptive analyses were conducted to determine if there were any significant demographic differences between the reimbursement and non-reimbursement groups. In this analysis, the outcome variable was survey completion. Operationally, we analyzed the probability of survey completion at three- and six-month follow-up time-points only since there was variation in participant knowledge of reimbursement at these time-points. Conversely, all participants completed the baseline assessment in order to be randomized into the study, and the reimbursement procedures were the same for both groups once the first participants reached the 12-month follow-up assessment window. Status of reimbursement was the primary explanatory factor. The other explanatory variable was time, measured as the number of months elapsed from baseline. An interaction term between time and reimbursement status was created to capture the changing pattern over time on survey completion. Education, gender, and race/ethnicity (Caucasian vs. Other) were rescaled to be centered at sample means and used as control variables in estimating the model parameters.

Given the binary outcome data, we applied the mixed-effects binary logit model [14] for the analysis. In light of analytic results of a preliminary data analysis, we used the random intercept logit model, assuming the effects of financial reimbursement on the logit of survey completeness to be fixed over time. The SAS PROC NLMIXED procedure (SAS Institute Inc., Cary, NC) was applied to compute the fixed and the random effects given its flexibility in estimating and predicting parameters in generalized linear mixed models [15]. With the specification of between-subjects random intercepts, time was treated as a continuous variable.

We predicted the probability of survey completion at three-month and six-month follow-up time-points for each reimbursement group by applying the best linear unbiased predictor. In predicting trajectories of survey completion probabilities, values of all the control variables were held at time-specific sample means. Finally, we plotted trajectories of survey completion probabilities to display its pattern of change over time for the two reimbursement groups.

3. Results

Participant characteristics are presented in Table 1. Survey completion rates across the follow-up time-points were high in both reimbursement status groups, with 82% or higher survey completion at both time-points. Results of the mixed-effects logit model demonstrate a significant main effect for reimbursement on survey completion (Table 2). Participants who received reimbursement were significantly more likely to complete follow-up assessments at both three- and six-month time-points, controlling for gender, education, and race/ethnicity (Table 3). At three-month follow-up, the probability of completing the study assessment was 98% for participants who received reimbursement, compared to 93% for participants who did not receive reimbursement. Similarly, the likelihood of completing the six-month follow-up assessment was 95% for participants who received reimbursement compared to 91% for participants who did not receive reimbursement. There was also a significant main effect for time (p < 0.01), such that participants were less likely to complete the study assessment at six-month compared to three-month follow-up, but no significant interaction between time and incentives was found (p = 0.15).

Table 1.

Participant characteristics.

	3-month follow-up				6-month follow-up
	No Reimbursement (N = 237)		Reimbursement (N = 429)		No Reimbursement (N = 85)		Reimbursement (N = 581)
Characteristic	N	%	N	%	N	%	N	%
Gender
Female	46	19%	81	19%	16	19%	111	19%
Male	191	81%	348	81%	69	81%	470	81%
Race
White	119	50%	199	46%	37	44%	281	48%
Other	118	50%	230	54%	48	56%	300	52%
Education level
High school	68	29%	135	31%	23	27%	180	31%
Some college	119	50%	206	48%	42	49%	283	49%
College degree	50	21%	88	21%	20	24%	118	20%
Survey Completion
Complete	204	86%	413	94%	70	82%	529	91%
Incomplete	33	14%	16	4%	15	18%	52	9%

Table 2.

Analytic results and summary measures for the mixed-effects logit model on survey completeness: STEPS-UP study (N = 666; df = 665).

Explanatory variable and effect measure	Regression coefficient	Standard error	t value	p value $>\left|t\right|$
Fixed Effects
Intercept	5.526∗∗∗	0.555	9.95	<0.01
Time (centered at month three)	−0.446∗∗∗	0.126	−3.54	<0.01
Incentives (1 = no incentives)	−1.894∗∗∗	0.469	−4.04	<0.01
Time (centered) × incentives	0.310	0.215	1.44	0.15
Education (centered)	0.707∗∗∗	0.147	4.80	<0.01
Male (centered)	0.701	0.477	1.47	0.14
White (centered)	0.059	0.372	0.16	0.87
Random Effects
Intercept	2.265∗∗∗	0.277	8.18	<0.01
−2 log likelihood	654.00

*** p-value < 0.01.

Note: The random effect of the intercept is parameterized by the standard error of the random effects.

Table 3.

Predictive probabilities.

	Reimbursement	No Reimbursement
	M (SD)	M (SD)
3-month time-point	0.98 (0.05)	0.93 (0.14)
6-month time-point	0.95 (0.11)	0.91 (0.16)

4. Discussion

This secondary analysis was a naturalistic study conducted as part of an effectiveness trial within a real-world military health care setting. Results from this research suggest that providing financial reimbursement to active duty Service members for participation in clinical trials modestly increases retention rates. At three-month follow-up, financial reimbursement was associated with a 5% increase in retention rates between participants who received reimbursement and those who did not. Similarly, financial reimbursement was associated with a 4% increase in retention rates at six-month follow-up. These results demonstrate a small yet significant effect of reimbursement on retention rates in a sample of active duty Service members. To the best of our knowledge, this is the first analysis of the impact of financial reimbursement on a Service member population participating in a clinical trial.

High recruitment and retention rates in clinical trials are necessary to achieve adequate power. Adequate power enables research to better account for confounding explanatory factors, detect meaningful differences between study groups, and produce more refined estimates of population parameters [8]. If a study is underpowered due to poor recruitment or significant attrition, investigators and funding agencies are unable to provide meaningful answers to important research questions [16]. Significant funds are devoted to the management of research programs within the DoD; in fiscal year 2017, the Congressionally Directed Medical Research Program initiated management of $1,117.1 million across 31 research programs [17]. However, many barriers exist that may impede high research follow-up rates in active duty Service members to include mobility of the population due to frequent moves, deployments, and training exercises, and strict work schedules that make it difficult to take time to participate in research [3]. Recent controlled trials of psychotherapy interventions with Service members have demonstrated rates of attrition at three- and six-month follow-up ranging from 27% to 49% [[5], [6], [7]]. Our results suggest that financial reimbursement may improve study completion rates in military research trials and thus may increase the probability of providing meaningful study results. Investigators, funding agencies, and other organizations may find these results useful when planning retention strategies and budgets for future studies.

Our findings are consistent with other longitudinal studies that have found 2%–13% increases in retention rates with the use of monetary incentives [9]. Findings from this analysis may be particularly relevant for clinical trials with smaller sample sizes, where a 4%–5% increase in retention rates with the use of reimbursement may have a substantial impact on increasing power. Indeed, a 4%–5% increase in participant retention may reduce bias, in line with the “five-and-twenty rule,” which states that generally, acceptable loss of follow-up rates in randomized trials often range between 5% and 20%, with less than 5% loss associated with little bias, and greater than 20% loss associated with potentially serious bias [18]. In studies with historically difficult populations to engage in research (e.g., Service members) or studies looking at low base rate events (e.g., suicide), a 4%–5% increase in retention may have a significant impact on successful study completion.

Many studies in the military rely upon the intrinsic motivation among Service members to contribute to an expanding knowledge base, and some researchers prefer not to use monetary incentives in their research. This may be appropriate in studies where researchers try to keep the conditions as “real-world” as possible (e.g., in pragmatic trials); in this context, offering monetary reimbursement to receive health care may limit the ability for studies to understand real barriers and facilitators to care. However, when participation is critical for ensuring adequate power to assess efficacy and effectiveness of new interventions, financial reimbursement may be essential to the study's success.

Limitations to the current findings need to be considered when interpreting the results. In general, the study achieved high assessment retention rates across all follow-up time-points. It is possible that a response rate ceiling was reached in this study, resulting in a potentially modest impact of reimbursement on overall retention rates. In other studies where reimbursement is the only retention strategy employed, it may have a different level of impact on research retention rates. Next, multiple retention strategies in addition to reimbursement were offered to study participants, including multiple assessment formats and types of reminders to complete assessments, expanded follow-up data collection windows, and a variety of research coordinator strategies to encourage participation. These additional strategies may have impacted overall retention rates, rather than just reimbursement alone. Other reviews have suggested that using multiple strategies have increased retention rates [9]. However, for this secondary analysis, we have a within-subjects design; both the reimbursement and non-reimbursement groups received the additional retention strategies mentioned above. Therefore, these multiple strategies were controlled across groups, meaning that the 4%–5% increase in retention rates may be the independent impact of incentives. Another limitation of this study is that we are unable to determine an optimal reimbursement amount needed to maximize retention rates. We also did not evaluate time to survey completion in this analysis, and are unable to determine whether individuals in the reimbursement group compared to the non-reimbursement group completed follow-up surveys earlier in the response window. Future analyses may take this into consideration, as it may have cost implications for future studies. Finally, evaluation of the impact of reimbursement was not a primary outcome of the study design; instead, due to progression of the trial over time, differences in reimbursement distribution were able to be analyzed post hoc. There may have been differences between those enrolled prior to incentives and those enrolled after the fact, and the differing retention rates may be a product of these group imbalances.

Despite these limitations, this study has several strengths. First, there is a dearth of information on the impact of reimbursement in research with military samples. Results from this study provide initial evidence of the impact of monetary reimbursement, suggesting a modest yet significant impact. It would add to the data if future studies continued to evaluate the impact of reimbursement to determine if these findings are consistent within other active duty populations participating in research. The design of the study is also a strength. The overall study is a multi-site RCT with a relatively large sample size, and the nature of the analyses presented here allow for a within-subjects design to assess the impact of reimbursement on follow-up retention rates. Examining the effects of reimbursement using a within-subjects design allows investigators to better control for potentially confounding factors such as site and treatment differences in order to isolate the potential impact of reimbursement. Additionally, the study used a graduated reimbursement schedule, starting with a $40 gift card for completing the eligibility assessment to a $55 gift card for completing the 12-month follow-up assessment. These rates were set high enough to compensate participants for the time and effort required to participate in study procedures, yet not deemed coercive so as to improperly influence participation. Previous studies have found retention rates are positively associated with increasing incentive amounts [9]. It would be useful if future research examined the amount and frequency of reimbursement necessary to improve retention rates. Cost benefit or cost effectiveness analyses may help examine whether the potential retention benefits and outcomes associated with providing reimbursement to clinical trial participants outweighs the costs.

5. Conclusions

Overall, obtaining high retention rates in research is important to ensure study results are valid and reliable. Offering financial reimbursement to active duty participants may be associated with higher follow-up completion rates. Findings from this trial suggest a potential benefit to modify current policies to allow reimbursement for research participation to active duty Service members. Given substantial funding invested into military trials, one concern is that the inability to reimburse Service members for research participation may compromise the successful completion of studies. Reimbursement policies for research with active duty Service members can ensure incentives are ethical and not coercive, and help dictate the value, frequency, and appropriateness of incentives in the context of ensuring researchers meet target recruitment and retention goals. More controlled research on the effects of financial reimbursement on recruitment and retention in military research is needed to substantiate the current results and further inform policy decisions about reimbursement benefits.

Conflict of interest disclosures

The authors have no conflicts of interest to declare.

Important disclaimers

The opinions expressed are those of the authors and do not necessarily reflect the views of the Department of Defense, Uniformed Services University of the Health Sciences, National Institutes of Health, the U.S. Government, or any other organization or agency, public or private.

Appendix A. Supplementary data

The following is the Supplementary data to this article: