Indications of Recruitment Challenges in Research with U.S. Military Service Members: A ClinicalTrials.gov Review

Abstract

Introduction

The success of military-relevant health research often depends on recruiting adequate numbers of U.S. military service members as research participants. Researchers have reported difficulties in recruiting service member research participants. Reviews of ClinicalTrials.gov, an online clinical trial registry of publicly and privately sponsored studies, have identified challenges in participant recruitment and barriers to study completion in various research populations. The purpose of this study was to identify indications of difficulty recruiting U.S. military service members as research participants based on data from study records in ClinicalTrials.gov.

Materials and Methods

Records of studies starting between 2005 and 2014 were collected from ClinicalTrials.gov and updated through January 2016. Three hundred and two studies that included ≥25% U.S. military service member research participants were (1) compared to a comparison group of 302 studies, each with <5% service member participants and (2) compared by the proportion of service member participants within studies in the military group (“many” ≥25% but <100% service members and “all” 100% service members). Groups were evaluated and compared for recruitment status; reasons for study withdrawal, termination, or suspension; achievement of ≥85% of the anticipated enrollment; and differences in achieving recruitment goals according to study sponsor.

Results

Twelve percent of studies in the military group had been withdrawn, terminated, or suspended; enrollment and funding problems were the most common reasons. The comparison group had 11% of studies withdrawn, terminated, or suspended; the most common reasons were enrollment problems and sponsor decision. All study groups had indications of difficulty adequately achieving participant enrollment goals. Among studies with known anticipated and actual enrollment, approximately half in both the military group (47.9%) and comparison group (50.3%) achieved ≥85% of the anticipated enrollment (p = 0.722). Half of studies with many service members and 44% of studies with all service members achieved ≥85% of the anticipated enrollment (p = 0.600). In comparing the many and all service member subgroups, significant differences were found in the median values for anticipated enrollment and actual enrollment, even when accounting for Bonferroni correction. Evaluations of mean values did not show a statistical difference between the military subgroups. There were no significant differences according to study sponsor (military, academic, Veterans Affairs, National Institutes of Health, nonprofit organization, or industry) for a study achieving or not achieving ≥85% of the anticipated enrollment.

Conclusion

This review supports anecdotal reports of difficulty recruiting service members as research participants. However, the findings also indicate that in many regards, there is not much difference in the difficulties recruiting service members versus other research participants. Findings suggest that it is often difficult to recruit research participants regardless of the specific population or type of study sponsor, and that studies with either many or all service member participants have similar achievement of recruitment goals. Findings in this study may be useful for those who design research that includes service members or for those who are apprehensive about including service members in their research.

INTRODUCTION

The success of military-relevant health research often depends on recruiting adequate numbers of U.S. military service members as research participants. While critical for successful research on military health issues, researchers have reported difficulties in recruiting service members as research participants.^1–3

Bush et al¹ described their experiences of difficulties recruiting service members as research participants, while noting a lack of empirical data documenting such difficulties, despite anecdotal evidence. In their search of literature from 2001 to 2012, they identified only 24 published randomized controlled trials that both included active duty service members and reported recruitment or attrition data. Almost all of the studies included attrition data but did not report enrollment rates.¹

Williams et al² reported difficulties, particularly delays, in conducting military research and recommended reforms in military human subjects protections that could potentially alleviate some of the reported problems. They discussed their experiences of unnecessary delays in military institutional review board reviews, which led to inadequate time to recruit participants within grant timelines and increased difficulty in recruiting participants due to staff changeover.

Braun et al³ presented a review of literature that identified recruitment challenges in research with U.S. military women. Their findings indicated that there is limited literature regarding recruitment and retention challenges in research with service members as research participants, and specifically with female service members as research participants.

One information source containing anticipated and actual participant enrollment data is ClinicalTrials.gov, a publicly available, online clinical trial registry of publicly and privately sponsored studies.⁴ Principal investigators (PIs) and study sponsors register their studies in ClinicalTrials.gov for several reasons, including to communicate transparently with the public,⁵ to comply with Food and Drug Administration regulations,⁶ and to meet journal recommendations.⁷

Recent reviews of ClinicalTrials.gov have identified challenges in participant recruitment and barriers to study completion.^8–11 However, no reviews of ClinicalTrials.gov have explored such challenges in research with U.S. military service members. The purpose of this study was to identify indications of difficulty recruiting U.S. military service members as research participants based on data from study records in ClinicalTrials.gov. Records of studies starting between 2005 and 2014 and with at least 25% U.S. military service members as research participants were compared both within the group, according to the proportion of service member participants, and outside the group, to a similar group of studies with no or very few U.S. military service members.

METHODS

Identification of Studies

A search of the ClinicalTrials.gov registry, described in detail elsewhere,¹² was performed to identify studies in which U.S. military service members participated. The search was limited to studies that were first received in ClinicalTrials.gov between January 1, 2003, and August 1, 2015, and included adult research participants (18–65 years of age). Search terms included military, active duty, soldier, sailor, airman, marine, guardsman, Air Force, Army, Marine Corps, Navy, National Guard, service member, deployment, combat, war, traumatic brain injury, post-traumatic stress disorder, and reintegration.

Each study was evaluated for three inclusion criteria: (1) The study started between January 1, 2005, and December 31, 2014; (2) The study required active involvement of participants; and (3) At least 25% of the study’s actual or intended participants were current members of any U.S. military service (Army, Navy, Air Force, and Marine Corps) and of any duty status (active duty, National Guard, and Reserve) at the time of enrollment.

After identifying studies for inclusion, the studies were further categorized according to the proportion of service member participants. Studies were categorized as having (1) many service member participants if at least 25% but less than 100% of participants were estimated to be U.S. service members or (2) all service member participants when information in a study’s record specified that 100% of participants were U.S. service members. The combined group of included studies with many and all service members will be referred to as the “military group.”

Three hundred and two U.S. studies with adult participants were randomly selected from studies identified in the search that did not meet inclusion criteria. These studies were used as a comparison group of studies that shared characteristics with the studies in the military group, but whose participants were less than 5% service members. The limit of less than 5% service members was used because it was usually not possible to confirm that a study had included no service members; however, it was possible to identify studies that likely included no or very few service members. The comparison group was matched to the military group by study start year, to have equal numbers of studies starting each year between 2005 and 2014, to account for research trends that may have occurred over this time frame, such as funding availability or interest in research participation.

Data Collection

For each study in the military and comparison groups, the study record in ClinicalTrials.gov was reviewed for data elements that could indicate difficulty recruiting participants. Since a study’s registry record is updated periodically by the PI or sponsor,¹³ records were reviewed for updates throughout this review process; any information updated as of January 31, 2016, was used for analysis.

Recruitment Status

Recruitment status of each study was assessed using ClinicalTrials.gov categories of participant recruitment status.¹⁴ Recruitment status categories and definitions are provided in Table I.

TABLE I.

Recruitment Status Definitions

Category	Definition
Not Yet Recruiting	Participant recruitment has not begun
Recruiting	Recruitment of participants in progress
Active, Not Recruiting	No longer recruiting or enrolling new participants; enrolled participants continue to be involved
Completed	The study has reached its anticipated end; participants are no longer actively involved
Suspended	Recruitment and enrollment have stopped early but may be resumed at some point
Terminated	At least 1 participant enrolled, the study stopped early, no additional recruitment or enrollment, and participants are no longer actively involved
Withdrawn	The study was stopped early and no participants were enrolled

Definitions adapted from ClinicalTrials.gov data element definitions.¹⁴

Reason for Withdrawal, Termination, or Suspension

For studies that were withdrawn, terminated, or suspended—in other words, stopped before reaching completion as planned—the reason was assessed. In ClinicalTrials.gov, the PI or sponsor may provide a short free-text description of up to 160 characters to explain the reason a study was stopped. However, providing a reason for stopping a study is not required.¹⁴ Reasons provided were grouped into 10 categories for analysis.

Anticipated and Actual Enrollment

Each study’s record was reviewed to determine both anticipated enrollment and actual enrollment. This was accomplished by using the Study History feature in ClinicalTrials.gov, which provides access to all changes that have been made to a particular study record since its creation.¹³ Actual enrollment data were not available for all studies, because some studies were still recruiting participants and some records of completed studies had not been updated with actual enrollment information. Some studies that were registered in ClinicalTrials.gov late (while already in progress or completed), and when the actual enrollment information was already known, the anticipated enrollment was not always provided.

Study Sponsor

For each study, the sponsor was categorized into one of six categories: military, academic, Veterans Affairs, National Institutes of Health, nonprofit organization, or industry.

Data Analysis

The units of analysis for this project are studies registered in ClinicalTrials.gov. The number of studies meeting inclusion criteria was thus fixed in advance. During study planning, it was estimated that there were at least 250 studies available for analysis in the military group. Assuming two-tailed statistical tests, α significance threshold = 0.05, statistical power β = 0.80, and at least 250 available studies, this project would be projected to have a minimum detectable effect size d >0.25 or a minimum detectable effect size r >0.125. These are effect sizes that Cohen¹⁵ characterizes as “small-to-medium.” True effect sizes smaller than these limits would not be detectable with the fixed available sample size of this project. Some of the analyses compare smaller subsets of the sample. The statistical sensitivity of these subsidiary analyses will be correspondingly smaller.

Descriptive statistics, including frequencies and percentages of categorical variables, were performed using SPSS Statistics, version 23 (IBM, Armonk, New York). Cross tabulations of two categorical variables were performed using two-sided Fisher’s exact test to calculate p value with level of significance α = 0.05. The effect sizes for the categorical analyses were reported as the odds ratio (OR). Student independent samples parametric t test was used to compare means. Mean effect size was calculated using means and SD. The nonparametric Mann–Whitney test was performed to compare two groups. Median effect size was calculated by rank-biserial correlation. Because over 60 analyses were performed on the same data, a Bonferroni correction was used. The adjusted statistical threshold required a p value of < 0.00076 to be considered statistically different.

For studies in which both the actual and anticipated enrollment numbers were known, studies were categorized as having adequately met participant enrollment goals if the final enrollment was at least 85% of the anticipated enrollment. This metric was based on that used by Carlisle et al¹⁰ in their ClinicalTrials.gov review of studies that did not achieve adequate enrollment or were terminated. The authors argued that it was reasonable to conclude that studies that attained less than 85% of the anticipated number of participants were compromised due to inadequate statistical power.¹⁰

RESULTS

A total of 302 studies were identified that included current U.S. military service member participants as defined by the inclusion criteria. Of these studies, 182 (60.3%) included many U.S. service member participants (at least 25% but not all) and 120 (39.7%) included all service member participants.

Recruitment Status

Recruitment status is reported in Table II. The only category of recruitment status that differed significantly between the military group and the comparison group was the category of studies “not yet recruiting”; there were significantly more studies in the military group that were “not yet recruiting” compared to the comparison group (n = 10 [3.3%] and n = 0 [0%], respectively, p = 0.002). However, this did not meet the Bonferroni adjusted statistical threshold required to be considered statistically different. There were no statistical differences between the military subgroups with many and all service members.

TABLE II.

Recruitment Status by Study Group

Recruitment Status N (% of Total Studies in Group)	Military Group (N = 302)	Comparison Group (N = 302)	p Valuea	ES (OR)	All Military Participantsb (N = 120)	Many Military Participantsc (N = 182)	p Valuea	ESd (OR)
Not Yet Recruiting	10 (3.3)	0	0.002e	11.339	6 (5.0)	4 (2.2)	0.203	2.342
Recruitingf	84 (27.8)	93 (30.8)	0.475	0.866	35 (29.2)	49 (27.0)	0.695	1.117
Active, Not Recruiting	45 (14.9)	30 (9.9)	0.084	1.588	21 (17.5)	24 (13.2)	0.325	1.397
Completed	127 (42.1)	146 (48.3)	0.141	0.775	45 (37.5)	82 (45.1)	0.234	0.732
Suspended	2 (0.66)	1 (0.33)	1.00	2.007	0	2 (1.1)	0.519	—
Terminated	18 (6.0)	24 (7.9)	0.424	0.734	9 (7.5)	9 (4.9)	0.457	1.558
Withdrawn	16 (5.3)	8 (2.6)	0.143	2.056	4 (3.3)	12 (6.6)	1.00	0.489

^aFisher’s exact test.

^b100% military service member participants.

^cAt least 25% but less than 100% military service member participants.

^dDirection of the effect size is given in terms of the all military participants group compared with the many military participants group.

^eSignificant at p < 0.05. Not significant using Bonferroni correction requiring p < 0.00076.

^fThis category includes studies “enrolling by invitation,” a ClinicalTrials.gov recruitment status in which only participants from a predetermined population are invited to participate.¹⁴

Reasons for Withdrawal, Termination, or Suspension

In the military group, 36 studies (11.9%) were withdrawn, terminated, or suspended, compared to 33 studies (10.9%) in the comparison group (Table III). The two most common reasons reported in the military group were enrollment problems (n = 11, 30.6%) and funding problems (n = 10, 27.8%). In the comparison group, the most common reasons reported for study withdrawal, termination, or suspension were enrollment problems (n = 8, 24.2%) and sponsor decision (n = 5, 15.2%). None of the differences were statistically detectable.

TABLE III.

Reasons for Study Withdrawal, Termination, or Suspension by Study Group

	Military Group (N = 302)	Comparison Group (N = 302)	p Valuea	ES (OR)	All Military Participantsb (N = 120)	Many Military Participantsc (N = 182)	p Valuea	ESd (OR)
Number of Studies Withdrawn, Terminated, or Suspended N (% of Total Studies in Group)	36 (11.9)	33 (10.9)	0.798	1.103	13 (10.8)	23 (12.6)	0.718	0.840
Reason for Withdrawal, Termination, or Suspensione
N (% of Total Withdrawn, Terminated, or Suspended Studies in Group)
Enrollment Problems	11 (30.6)	8 (24.2)	0.642	1.375	4 (30.8)	7 (30.4)	1.00	1.016
Funding Problems	10 (27.8)	3 (9.1)	0.066	3.846	3 (23.1)	5 (21.7)	1.00	1.080
No Reason Provided	10 (27.8)	4 (12.1)	0.139	2.788	4 (30.8)	6 (26.1)	1.00	1.259
Methods Problems	2 (5.6)	1 (3.0)	1.00	1.882	0	2 (8.7)	0.525	—
Product/Device Problems	2 (5.6)	1 (3.0)	1.00	1.882	0	2 (8.7)	0.525	—
Administrative Problems	1 (2.8)	3 (9.1)	0.343	0.286	1 (7.7)	0	0.361	—
PI Left Facility	1 (2.8)	4 (12.1)	0.186	0.207	1 (7.7)	0	0.361	—
Sponsor Decision	1 (2.8)	5 (15.2)	0.097	0.160	0	1 (4.3)	1.00	—
Decided Not to Start Study	0	2 (6.1)	0.225	—	0	0	—	—
Futility	0	3 (9.1)	0.104	—	0	0	—	—

^aFisher’s exact test.

^b100% military service member participants.

^cAt least 25% but less than 100% military service member participants.

^dDirection of the effect size is given in terms of the all military participants group compared with the many military participants group.

^eFor three studies (two from the military group and one from the comparison group), two reasons for withdrawal or termination were provided, and both reasons are included in this table. For all three studies, funding problems was one of the two reasons.

Anticipated and Actual Enrollment

Both anticipated and actual enrollment numbers were available for 140 studies (46.4%) in the military group and 143 studies (47.4%) in the comparison group (Tables IV and V). Among studies with known anticipated and actual enrollment, approximately half in both the military group (n = 67, 47.9%) and comparison group (n = 72, 50.3%) achieved at least 85% of the anticipated enrollment (p = 0.722).

TABLE IV.

Comparison of Anticipated and Actual Enrollment Numbers by Study Group

	Military Group (N = 140)	Comparison Group (N = 143)	P Valuea	ES (OR)	All Military Participantsb (N = 52)	Many Military Participantsc (N = 88)	P Valuea	ESd (OR)
Percentage of Total Studies in Group With Known Anticipated and Actual Enrollment Numberse	46.4	47.4	0.870	0.961	43.3	48.4	0.411	0.817
Studies With Actual Enrollment Meeting Recruitment Goals
N (% of Studies in Group With Anticipated and Actual Enrollment)
Actual Enrollment ≥85% of Anticipated Enrollment	67 (47.9)	72 (50.3)	0.722	0.905	23 (44.2)	44 (50.0)	0.600	0.793
Actual Enrollment ± 5% of Anticipated Enrollment	44 (31.4)	37 (25.9)	0.357	1.313	13 (25.0)	31 (35.2)	0.259	0.613
Studies With Actual Enrollment Below Anticipated Enrollment
N (% of Studies in Group With Anticipated and Actual Enrollment)
<85% of Anticipated Enrollment	73 (52.1)	71 (49.7)	0.722	1.105	29 (55.8)	44 (50.0)	0.600	1.261
<75% of Anticipated Enrollment	60 (42.9)	59 (38.6)	0.810	1.068	23 (44.2)	37 (42.0)	0.860	1.093
<50% of Anticipated Enrollment	38 (27.1)	33 (23.1)	0.493	1.242	13 (25.0)	25 (28.4)	0.699	0.840
<25% of Anticipated Enrollment	16 (11.4)	15 (10.5)	0.851	1.101	4 (7.7)	12 (13.6)	0.411	0.528
<10% of Anticipated Enrollment	9 (6.4)	8 (5.6)	0.807	1.159	2 (3.8)	7 (8.0)	0.484	0.463

^aFisher’s exact test.

^b100% military service member participants.

^cAt least 25% but less than 100% military service member participants.

^dDirection of the effect size is given in terms of the all military participants group compared with the many military participants group.

^eWithdrawn or terminated studies with known actual and anticipated enrollment numbers were included if the reason provided was problems with enrollment. Terminated studies not providing a reason for termination were included because at least one participant must have enrolled to be categorized as terminated. Withdrawn studies not providing a reason for withdrawal were not included because there was no way to verify that the study had ever been initiated (studies classified as withdrawn must not have enrolled participants as defined by ClinicalTrials.gov¹⁴).

TABLE V.

Descriptive Statistics for Studies With Anticipated and Actual Enrollment Numbers

	Military Group (N = 140)	Comparison Group (N = 143)	P Valuea	ES	All Military Participantsb (N = 52)	Many Military Participantsc (N = 88)	P Valuea	ESd
Anticipated Enrollment
Mean (SD)	700.0 (3,704.6)	180.9 (302.3)	0.101	0.098 (r) 0.198 (d)	1,689.4 (5,974.2)	115.4 (272.4)	0.063	0.183 (r) 0.372 (d)
Mediane	78.0	80.0	0.848	−0.013 (r)	171.0	51.0	<0.0005f	0.565 (r)g
Range	(5,36,000)	(10,2,000)	—	—	(15,36,000)	(5,2,380)	—	—
Actual Enrollment
Mean (SD)	417.4 (2,599.1)	154.2 (281.6)	0.236	0.071 (r) 0.142 (d)	1,012.3 (4,221.5)	65.8 (96.3)	0.112	0.157 (r) 0.317 (d)
Median	51.0	64.0	0.765	−0.021 (r)	86.0	35.5	<0.0005f	0.540 (r)g
Range	(0, 30,209)	(0, 2,011)	—	—	(4, 30,209)	(0, 740)	—	—
Difference Between Actual and Anticipated Enrollment
Mean (SD)	−282.7 (2,165.7)	−26.7 (92.6)	0.165	−0.083 (r) −0.167 (d)	−677.0 (3,527.9)	−49.7 (224.0)	0.206	−0.125 (r) −0.251 (d)
Median	−9.5	−9.0	0.135	−0.102 (r)	−17.0	−5.5	0.025h	−0.225 (r)
Range	(−24,859, 1,625)	(−360, 536)	—	—	(−24,859, 1,625)	(−2,066, 80)	—	—
Percent Difference Between Actual and Anticipated Enrollment
Mean (SD)	−25.1 (40.8)	−19.9 (43.2)	0.296	−0.062 (r) −0.124 (d)	−27.3 (32.9)	−23.8 (44.9)	0.605	−.044 (r) −0.089 (d)
Median	−17.2	−15.0	0.262	−0.077 (r)	−19.7	−15.9	0.641	−0.047 (r)
Range	(−100, 140)	(−100, 165)	—	—	(−99.4, 60.2)	(−100, 140)	—	—

^aStudent independent samples t test used to compare means. Mann–Whitney test used to compare medians.

^b100% military service member participants.

^cAt least 25% but less than 100% military service member participants.

^dDirection of the effect size is given in terms of the all military participants group compared with the many military participants group.

^eBoth the median and mean are reported to show where unusually large studies skewed the distribution.

^fSignificant using Bonferroni correction requiring P < 0.00076.

^gr = 0.50 is a large effect size.¹⁵

^hNot significant using Bonferroni correction; significant at P < 0.05.

In comparing the military subgroups with many and all service member participants, both the median and mean were reported to show where a subset of unusually large studies skewed the distribution. Significant differences were found in the median values for anticipated enrollment and actual enrollment, even when accounting for the Bonferroni correction (Table V). The median difference between actual and anticipated enrollment was significantly lower (indicating a greater deficit in number of participants) in the group with all service members compared to the group with many service members, but was not significant when accounting for Bonferroni correction. The median percent differences between actual and anticipated enrollment were not statistically detectable.

Evaluations of mean values for anticipated enrollment, actual enrollment, difference between actual and anticipated enrollment, and percent difference between actual and anticipated enrollment did not show a statistical difference between the groups with many and all service members. Evaluating the percent difference between actual and anticipated enrollment normalized for the disproportionately larger number of participants in studies in the group with all service members. No difference was found between the groups.

Study Sponsor

For studies in which anticipated and actual enrollment numbers were available, there were no significant differences according to 6 categories of study sponsors for a study achieving or not achieving at least 85% of recruitment goals when comparing the military group to the comparison group. Similarly, there were no significant differences according to study sponsors when comparing the military subgroups with many and all service member participants.

DISCUSSION

To identify indications of difficulty recruiting U.S. military service members for research, a systematic review of study records in ClinicalTrials.gov was performed for studies that included at least 25% service member participants. Findings provide additional evidence of difficulty recruiting U.S. military service members as research participants. However, the indications of challenges in participant recruitment among studies with U.S. service member research participants were similar to a comparison group of studies with no or very few service member participants. Findings also suggest that there is not a significant difference in difficulty recruiting U.S. service member participants when studies include all service members or include many but not all service members.

Recruitment Status

The only significant difference found between the military and comparison groups in this study was that of the “not yet recruiting” status, in which more studies in the military group were not yet recruiting. However, when considering Bonferroni correction, the statistical threshold for significance was not met. In addition, this difference in recruitment status does not necessarily indicate difficulty recruiting service members, as it entails the period before participant recruitment. It could indicate problems with delays in starting recruitment in the military group, which could be substantial enough to carry over into the recruitment phase and to cause, or at least be perceived to cause, difficulty in recruitment. Other possible explanations are differences in ClinicalTrials.gov registration patterns between military and nonmilitary sponsors, such as registering studies before receiving institutional review board approval, when participants could not yet be enrolled in a study, or batching registration of several studies at one time.

Reasons for Study Withdrawal, Termination, or Suspension

Difficulty with participant enrollment was the most common reason provided for withdrawal, termination, or suspension of studies regardless of study group. When considering only terminated studies, the military group had a similar, slightly higher proportion of studies terminated due to enrollment problems (44.4%) compared to several other ClinicalTrials.gov reviews reporting rates of trial terminations due to poor enrollment. In the Carlisle et al¹⁰ review, 43% of terminated studies were due to poor enrollment, 39% in the Williams et al¹¹ review, and 38% in the Cepeda et al⁹ review of analgesic trials. The present study’s comparison group had a lower proportion of terminated studies attributed to problems with enrollment (25%) than all of the aforementioned studies. However, Bernardez-Pereira et ⁸ found that among terminated cardiovascular studies, 53.6% were attributed to poor accrual, a higher proportion than the military group in this study. It is important to note that the other ClinicalTrials.gov reviews included a wide variety of conditions and populations. They are offered here to help show how the number of terminated studies due to enrollment problems in this review compares to those in other reviews of ClinicalTrials.gov.

Many studies in the military group that were withdrawn, terminated, or suspended did not provide a reason. This raises concern for two reasons. First, ClinicalTrials.gov provides a platform for researchers to communicate with the public. Although providing a reason for study withdrawal, termination, or suspension is not required by ClinicalTrials.gov¹⁴, by not doing so, the PI and sponsor miss an opportunity for transparency. Second, enrollment and funding problems were the most commonly reported reasons for studies in the military group being stopped before reaching completion as planned, and it is reasonable to assume, were reasons for several of the withdrawals, terminations, or suspensions that had no reason reported. Having such information documented would help researchers evaluate the challenges encountered in research with service members.

Anticipated and Actual Enrollment

Many studies in this review, in both the military and comparison group, did not adequately meet recruitment goals. Among the studies in each group for which anticipated and actual enrollment numbers were known, only about half achieved at least 85% of the anticipated enrollment. Studies in all groups clearly had difficulty enrolling the anticipated number of participants, while the group with all service member participants performed the worst, with the lowest percentage of studies (44.2%) that achieved at least 85% of anticipated enrollment.

For comparison, in their review of unsuccessful participant accrual, Carlisle et al¹⁰ found that out of 2,577 trials registered in ClinicalTrials.gov that met their inclusion criteria, 1,889 (73.3%) enrolled at least 85% of anticipated participants. Although the current study was not designed to replicate the Carlisle et al study, and therefore the comparison is not exact, the studies in this review were clearly less successful at achieving recruitment goals.

The subgroup of studies with all service member participants included several studies with enormous recruitment goals, some seeking several thousand participants. It is unknown if these anticipated enrollment numbers were realistic and based on power analyses. It is possible that these were unrealistically high enrollment goals based on investigators’ optimism or inexperience in recruiting research participants.

Study Sponsor

The propensity for studies to meet at least 85% of the recruitment goals does not appear to be affected by study sponsor, at least not when evaluating by broad categories such as military or academic sponsors, as used in this analysis. It is possible that within these large categories, certain components have greater success meeting recruitment goals. For instance, the category of military sponsor was not further differentiated by medical centers and research commands, or by service, and such entities could have differences in accomplishing recruitment goals.

Limitations

This study has several limitations. First, despite vigilance in regularly reviewing the study records for updated information, due to the volume of studies in the review, it is possible that some content changed by PIs or sponsors may have been overlooked. This type of oversight would result in the review including original or earlier record content rather than more recent record data.

Second, recruitment status and enrollment data are provided by the PI or sponsor,⁴ and if an update has not been provided, the data in ClinicalTrials.gov may not reflect the actual study status. Of studies in this review that were still in progress (n = 145, 48.0% in the military group, and n = 123, 40.7% in the comparison group), 37 (25.5%) studies in the military group and 27 (22.0%) studies in the comparison group had not been updated with new information between January 1, 2015, and January 31, 2016.

Third, the ability to make inferences using statistics may be limited by sample size, as is true in every study. Some of the groups, such as the 2 subgroups of military studies and studies for which anticipated and actual enrollment numbers were known, did not have 250 studies for some of the tests, as we had estimated when calculating statistical power. This may have left some tests under powered, but in light of the large effect size/small p values on the Mann–Whitney test, we think this is unlikely. However, the fact that some tests did not show a difference between groups when the subset samples were smaller, does not necessarily mean that there is not a difference.

A fourth limitation of this review is that the studies in the comparison group may have been too similar to the studies in the military group to detect differences in participant recruitment. For example, over one-quarter (27.8%) of the studies in the comparison group recruited either exclusively veterans as participants or veterans along with other groups of participants. Veterans and current military service members may have similarities in terms of research participation, such as reasons for participating in research or ways in which they become aware of research opportunities. It is important to note that this study was not designed to compare research with current service members and veterans. The group of military studies thus may not have been as different from the group of comparison studies as in other ClinicalTrials.gov reviews with comparison groups, such as the Pasquali et al¹⁶ review in which pediatric trials were compared to nonpediatric trials.

CONCLUSION

Recruiting U.S. military service members as research participants is challenging. This review supports, with empirical data, researchers’ anecdotal reports of difficulty recruiting military service members as research participants. However, the findings also indicate that in many regards, there is not much difference in the difficulties recruiting military service members versus other research participants. Studies with no or very few service members did not significantly surpass studies with service members in terms of accomplishing recruitment goals.

If there is not much difference in the achievement of recruitment goals between military and nonmilitary studies, it is worthwhile to consider if methods that are known to improve recruitment of other populations of research participants could be implemented as effectively in studies recruiting service members.

Findings in this study may be useful for those who design research that includes service members or for those who are apprehensive about including service members in their research. Findings suggest that it is often difficult to recruit research participants regardless of the specific population or type of study sponsor, and that studies with either many or all service member participants have similar achievement of recruitment goals. Studies that have achieved recruitment goals and studies that have failed to meet recruitment goals should be critically evaluated to identify and quantify the linking of specific methods and circumstances that hinder or support recruitment. Identifying effective strategies for reducing challenges in recruiting service member research participants, both before starting recruitment and throughout recruitment, warrants further consideration and investigation. This should also include appraisal of existing literature to identify effective methods for recruiting other populations of research participants that may be incorporated into studies which include service members.