Abstract
Background
Pilot studies are essential in determining if a larger study is feasible. This is especially true when targeting populations that experience stigma and may be difficult to include in research, such as people with HIV. We sought to describe how pilot studies have been used to inform HIV clinical trials.
Methods
We conducted a methodological study of pilot studies of interventions in people living with HIV published until November 25, 2020, using Medline, Embase, and Cochrane Controlled Register of Trials (CENTRAL). We extracted data on their nomenclature, primary objective, use of progression criteria, sample size, use of qualitative methods, and other contextual information (region, income, level, type of intervention, study design).
Results
Our search retrieved 10,597 studies, of which 248 were eligible. The number of pilot studies increased steadily over time. We found that 179 studies (72.2%) used the terms “pilot” or “feasibility” in their title, 65.3% tested feasibility as a primary objective, only 2% used progression criteria, 23.9% provided a sample size estimation and only 30.2% used qualitative methods.
Conclusions
Pilot studies are increasingly being used to inform HIV research. However, the titles and objectives are not always consistent with piloting. The design and reporting of pilot studies in HIV could be improved.
Supplementary Information
The online version contains supplementary material available at 10.1186/s40814-021-00934-9.
Keywords: Pilot study, Feasibility study, Feasibility outcome, Progression criteria, Key population, HIV
Introduction
Wastefulness in medical research is a major concern for researchers and funders and has been estimated to be at 85% of research investment [1]. There are several contributors to this waste. These include researchers not asking relevant questions, study results being inaccurately reported, and the inappropriate use of study design [2]. Recent work has demonstrated that pilot studies are very effective in reducing waste [3]. It was found that by employing pilot studies, the UK’s National Institute for Health Research’s (NIHR) Research for Patient Benefit (RfPB) Program saved approximately £20m, as otherwise non-feasible studies would have been conducted [3]. Pilot studies are especially useful in fields where participants are difficult to recruit and retain. However, despite the recognized value of pilot studies, there is still considerable confusion surrounding what constitutes a pilot study, how they should be designed, and how researchers decide whether they should proceed with the full study [4].
A pilot study is often described as a scaled down version of a larger study with feasibility, the assessment of the ability to conduct the full-scale study, as the primary goal [5]. Given the numerous challenges of recruiting and retaining participants in HIV research, pilot studies are particularly important in this field [6–15]. In this context, pilot studies could help researchers understand how they can adjust their procedures and reduce waste, especially when working with populations that experience difficulties, including stigma and discrimination [3]. People with HIV may belong to other key populations and face additional social stigma (e.g., men who have sex with men [MSM], commercial sex workers [CSWs], or people who inject drugs [PWID]), making participation in research more difficult [8–12, 16]. Studies have found that certain subpopulations (e.g., women, PWID, and African, Caribbean and Black (ACB) peoples) have dropout rates in research studies ranging from 30 to 50% [17–19].
The primary goal of this study was to describe the design, analyses, and reporting of pilot studies in HIV. The main outcomes of interest were the following:
The nomenclature of pilot studies (“pilot” or “feasibility” in the title)
Their declared primary objective (feasibility or effectiveness) and primary outcome
The use of progression criteria (criteria that would inform the decision to move to a larger trial)
Sample size estimation or justification (a description of why the sample size was chosen)
Use of qualitative methods (inclusion of qualitative assessments to inform feasibility)
The inclusion of key populations as defined by United Nations Program on HIV/AIDS (UNAIDS) [20–23]
Outcomes 1-5 have been identified as common shortcomings in pilot studies [6–15, 24–35]. The inclusion of key populations would indicate that a more diverse sample of people with HIV was included.
Methods
We conducted a methodological study of pilot studies in the HIV literature as per the guidelines reported by Murad and Wang for reporting meta-epidemiological research [36].
Ethics
This study used publicly available secondary data and therefore ethics review was not required.
Criteria for inclusion
All pilot studies of interventions conducted exclusively in people with HIV and published in English were eligible. We included randomized and non-randomized studies, using mixed or quantitative methods with at least one feasibility outcome [4]. Outcomes were deemed to be feasibility outcomes if they fit into a category outlined by Thabane et al. [4]. These categories include (1) assessing the processes involved in the study, (2) evaluation of resources required for the study, (3) management of potential human and data management problems, and (4) assessment of intervention safety, dose response, and variance of effect.
Search method for identifying pilot studies
We conducted an exhaustive search of the following databases: Medline, Embase, and the Cochrane Central Register of Controlled Trials (CENTRAL). These databases were searched from inception to November 25, 2020. Our search strategy was developed in collaboration with a librarian at the library services of the McMaster Health Sciences Central Library. The key concepts included in the search were “pilot,” “feasibility,” “proof-of-concept,” “exploratory,” “preliminary,” and “HIV.” The complete search strategies are included in the supplementary materials.
Screening and data extraction
We compiled the references and removed duplicate citations using Endnote X9 reference manager software [37]. We screened the remaining references first by their title and abstract and then by examining their electronic full texts. Both screening steps were done in duplicate by two independent reviewers using the Covidence data management platform for systematic reviews provided by McMaster University [38]. The reviewers attempted to resolve discrepancies by discussion and included a third reviewer if consensus could not be reached.
Data from included references were extracted using a piloted data-extraction form on RedCap [39]. Basic bibliometric information extracted from the studies included the following: the first author’s last name, study title, year of publication, journal of publication, and country of study (both region and income level). Region was determined using the regional groupings definitions provided by the World Health Organization (WHO), and income level was determined as per the World Bank Criteria [40, 41]. Other information collected included presence of a feasibility/pilot-identifying term in the title, study objectives, whether feasibility was a primary outcome, the use of progression criteria, a sample size justification, qualitative methods and study design, intervention type, and the inclusion of key populations (as defined by the UNAIDS and WHO [20–23]); key populations for which data were collected included (1) PWID, (2) MSM, (3) incarcerated populations, (4) CSW, (5) pregnant women, (6) children, (7) youth, (8) indigenous people, (9) ACB people, (10) women, and (11) transgender people. Progression criteria, having a feasibility related primary outcome and key labeling of pilot study status, are key characteristics of a pilot study. This information was extracted as these study characteristics have been found to impact study outcomes [42, 43]. We contacted authors via email to clear up ambiguity or to collect missing data.
Analysis
We conducted a descriptive analysis and reported counts and percentages for categorical variables and median (minimum, maximum) for continuous variables.
Results
Results of search
Our search returned 10,597 articles for title and abstract review. Of these, 536 were retrieved for full-text review. Only 248 articles met our eligibility criteria. Figure 1 is a flow chart of our screening and selection procedures.
Fig. 1.
Flow chart of study screening and selection
Characteristics of included studies
The 248 studies included in our review were published between 1998 and 2020, with a steady increase over time (Fig. 2). Less than half of the included studies were randomized (108/43.5%). The majority investigated a pharmaceutical intervention (227/91.5%); were conducted in the Americas (137/55.2%) or Africa (71/28.6%). Further characteristics are reported in Table 1.
Fig. 2.
Yearly trends in publication of pilot studies in the HIV literature
Table 1.
Baseline characteristics of included studies
| Characteristics | Statistic (n = 248) |
|---|---|
| Design, n (%) | |
| Randomized control trial | 108 (43.5) |
| Non-randomized trial | 140 (56.5) |
| Intervention type, n (%) | |
| Pharmaceutical | 21 (8.5) |
| Non-pharmaceutical | 227 (91.5) |
| Number of study sites, n (%) | |
| Single site | 143 (57.7) |
| Multiple sites | 105 (42.3) |
| Region, n (%) | |
| Africa | 71 (28.6) |
| Americas | 137 (55.2) |
| Southeast Asia | 11 (4.4) |
| Europe | 16 (6.5) |
| Western Pacific | 8 (3.2) |
| Mixed region | 5 (2) |
| Country income level, n (%)a | |
| High income | 151 (60.9) |
| Upper-middle income | 37 (14.9) |
| Lower-middle income | 25 (10.1) |
| Low income | 33 (13.3) |
| Mixed income | 2 (0.8) |
| Qualitative methods, n (%) | 75 (30.2) |
| Sample size: median (min, max) | 40 (3, 8794) |
| Key population, n (%) | |
| Contain any key population | 134 (54) |
| Specific key populations, n (%) | |
| Injection drug User | 4 (1.6) |
| MSM | 20 (8.1) |
| Incarcerated populations | 0 (0) |
| Commercial sex workers | 1 (0.4) |
| Pregnant women | 11 (4.4) |
| Children | 5 (2) |
| Youth | 28 (11.3) |
| Indigenous | 0 (0) |
| African, Caribbean, Black | 77 (31) |
| Women | 24 (9.1) |
| Transgender | 3 (1.2) |
| Progression criteria prespecified, n (%) | 5 (2) |
| Trial outcome, n (%) | |
| Proceed to larger study | 65 (26.2) |
| Do not proceed | 183 (73.8) |
Outcomes
Nomenclature
This has been clarified. New sentence: “Most studies were easily identifiable as pilot or feasibility studies, with 179 studies (72.2%) including the terms pilot, feasibility, or a feasibility outcome in the study title to denote the pilot status.” Feasibility outcomes were used to denote pilot status in 27 of these studies (15.1%). This was often done with the feasibility outcome “acceptance.” For example, a study was titled “Acceptability of a trial of vaginal progesterone for the prevention of preterm birth among HIV-infected women in Lusaka, Zambia: A mixed methods study” [44]. The remaining 69 studies (27.8%) had no indication of their pilot nature in the title.
Objectives
Study feasibility objectives were often clearly stated in the beginning or at the conclusion of the introduction. The primary objective was feasibility in 162 studies (65.3%). The remaining studies had goals centered around informing the sample size of the larger study, assessing efficacy, intervention development and to assess the reliability of a measure, with feasibility treated as a secondary outcome.
Outcomes
Feasibility outcomes were used as a primary outcome in 157 (63.3%) studies. We found that the feasibility outcomes assessed in the studies in this review could be grouped into 11 categories in total (Table 2). The most common outcomes were acceptance and retention of participants (180/72.6%; 135/54.4%, respectively), as well as evaluating participant enrolment (106/74.6%) and compliance (131/52.8%) to the intervention and study procedures. Trialists also often sought feedback (90/36.3%), primarily from participants and occasionally from study staff. Miscellaneous feasibility outcomes were also assessed and defined in 41 studies. The most common of these were implementation (34/13.7%), intervention initiation (6/2.4%), and consent rate (1/0.4%). The same outcome was defined differently in most instances, see Table 2. The CONSORT extension for pilot studies distinguishes between primary and secondary feasibility outcomes. The former are outcomes that inform the decision about progressing to a full RCT. Secondary feasibility outcomes are those related to patient centered outcomes [45].
Table 2.
Feasibility outcomes and definitions
| Outcomea (n/%) | Definition |
|---|---|
| Acceptance (180/72.6%) |
a) Participant satisfaction with the intervention and study procedures (165/91.7%) b) Participant recruitment or enrolment rate (10/5.6%) c) Intervention completion (16/8.9%) d) Participant retention (11/6.1%) e) Participant feedback (4/2.2%) f) Intervention usability (2/1.1%) |
| Contamination (2/0.8%) | The proportion of participants that deviated from their allocated intervention and partook in the alternative intervention |
| Complianceb (131/52.8%) |
a) Participant attendance of any intervention sessions or simply adherence to assigned intervention (27/20.1%) b) The number of sessions attended and engagement with the intervention (110/82.1%) |
| Data completion (9/5%) | The proportion of data expected to be collected which was not |
| Enrolment (106/42.7%) |
a) The proportion of eligible participants who consented to join the study (95/90.5%) b) The proportion of participants which were recruited and randomized (9/8.5%) c) The proportion of participants recruited, consented and were randomized (2/1.9%) |
| Feedback (90/36.3%) | What participants and study staff thought of the intervention, study procedures and their time in the trial |
| Fidelity (37/14.9%) | The ability of the trialists to adhere to study protocol |
| Randomization (11/4.4%) | The ability to successfully randomly allocate participants to the different arms in a trial |
| Retention (135/54.4%) | The proportion of participants which remained in the study till the primary endpoint (either end of the intervention or a set follow-up period) |
| Resources (16/6.5%) | An evaluation of the resources required to conduct the study |
| Timeliness of intervention (2/0.8%) | Assessment of the ability to administer the intervention in the prespecified time |
| Other (34/13.7%) |
a) Implementation—the ability to deliver the intervention to participants (17/65.38%) b) Initiation—the proportion of eligible participants which were recruited, consented to join the study and actually began using or were administered the intervention (6/23.07%) c) Consent rate—the proportion of eligible patients which consented to joining the study (1/3.8%) |
aStudies may possess several feasibility outcomes
bParticipant acceptance was often defined and assessed in multiple ways in the same study
Samples size estimation/justification
A sample size estimation was provided in 59 studies (23.8%). Forty-two of these studies (71.1%) provided an appropriate justification. Sample size justifications could be grouped into 6 categories. The most common justification was the use of a conventional sample size calculation with the intervention effect size to calculate a suitable sample size (20/33.9%). Researchers also equally relied on similar studies (6/10.2%) and on recommendations in the literature (6/10.2%). Three (5.1%) studies determined their sample size based on the resources available to conduct the study. Two (3.4%) studies used a proportion of the sample size of the larger study to justify their estimation. Finally, miscellaneous justifications were provided in 5 (10.2%) studies. For example, Tsima et al. justified their sample size using their estimation of expected recruitment rate [46].
Progression criteria
Only 5 (2%) of the included studies mentioned the progression criteria for their pilot studies. These studies were all RCTs published in 2020. Of these 5 studies, 3 (60%) were conducted in African countries. Of the 3 conducted in African countries, 2 (40%) were in low-middle-income countries and the other (1/20%) conducted in an upper-middle-income country. The other two were conducted in Europe and America.
Qualitative methods
Qualitative methods were included in only 75 studies (30.2%). Analysis was primarily conducted using data collected from participants. However, some studies also collected data from staff as well (25/33.33%).
Key populations
Close to half of the studies included a key population (134/54.0%). The complete composition of the patient sample is found in Table 1. ACBs represented the largest subpopulation among studies including key populations at 57% (n = 77). Only 24 (17.9%) of studies sought to purposely recruit women. Youth (28/17.2%) and MSM (20/14.9%) made up the remaining prominent vulnerable populations present. Other vulnerable populations present were PWID, CSW, pregnant women, children, and transgender women.
Discussion
To the best of our knowledge, this is the first methodological study of pilot studies in the HIV literature. We found that although pilot studies are becoming increasingly common in the HIV literature, there are considerable gaps in how they are labeled, designed, and how their findings are reported.
It is important to clearly label pilot studies and to use the correct terminology when doing so. Making pilot studies easier to identify helps inform readers that the primary goal of the study is to assess feasibility. We found that most authors (179/72.2%) labeled their studies clearly in the title.
Other reviews have found similar results. In two reviews of pilot studies in the cluster RCT literature and of pilot studies in the Clinical Rehabilitation journal, 83% and 87% of studies contained the terms pilot or feasibility in their title, respectively [24, 25]. The latter found that more than half of the studies used the pilot and feasibility terms interchangeably [25].
Our findings regarding authors replacing the primary feasibility objectives of their pilot studies, with other outcome(s) unrelated to feasibility, are in line with those of other authors [25–27]. The primary objective of a pilot or feasibility study must be to assess the feasibility of a larger study. As a result, assessing efficacy in a pilot study is inappropriate as this is not the primary goal of pilot studies and they are not powered to do so. However, smaller, non-pilot studies are important as they are important in hypothesis generation and in challenging widely held beliefs and common practices [47]. These studies should be encouraged and labeled properly as they also have a place in the literature [47]. The CONSORT extension for pilot studies of RCTs reinforces the requirement that pilot RCTs must have feasibility as their primary outcome by requiring feasibility outcomes be pre-specified and clearly defined [45].
Pilot studies must outline how authors intend to use their findings to inform future steps with pre-specified progression criteria [4]. However, only 5 (2%) studies assessed in this review included such criteria. Not reporting progression criteria is problematic, as we are unable to evaluate the criteria which the authors used to base their decision. A recent methodological review found that only 19.8% of studies included progression criteria [28]. Meanwhile, a review of cluster RCT pilot studies found that 89% of studies specified progression criteria [27]. However, the latter found that only 17% justified the criteria [27].
Currently, the CONSORT extension for pilot studies of RCTs requires reporting progression criteria, if applicable [45]. However, progression criteria are a key requirement of pilot studies, and this criterion should be strengthened to require the specification of the prespecified criteria. Progression criteria would improve the interpretability of the study [45]. The pilot studies assessed lacked both estimations and justifications for their sample sizes. Only 23.8% (n = 59) of studies had estimations for their sample size. However, even among these studies, 28.8% (n = 17) did not justify them. Chan et al. obtained similar results as they found that only 44% of pilot cluster RCTs justified for their sample size (27). As pilot studies do not aim to test hypotheses, formal power considerations are not necessary. However, it is still necessary to justify the sample size selected. As for now, guidance is only available for RCTs. As a result, concerns relating to progression criteria pertain more to pilot trials and are not always relevant for all pilot studies.
Several approaches can be used, such as targeting a percentage of the larger study’s sample size [48], having a set minimum number of participants per arm [48, 49], and using a stepped approach determined by standardized effect sizes [50]. Viechtbauer et al. have also proposed a sample size equation using the probability of a specific problem occurring during the trial to determine a sample size [51].
There have been recent calls for greater inclusion of qualitative methods in pilot studies [52, 53]. The use of qualitative methods has been shown to help refine study procedures, including optimizing recruitment and retention [29, 30]. By incorporating these methods in pilot studies, investigators are able to set realistic targets, craft pragmatic procedures, and ask and answer a wider range of questions, while gaining granular detail [53, 54]. As mentioned above, identification of pilot studies may be challenging given sub-optimal labeling. More so, identification of feasibility and pilot studies containing qualitative methods may be harder to identify as the qualitative components may be reported separately. These qualitative papers are harder to find if their titles do not contain the terms “pilot” or “feasibility.”
The HIV patient population is diverse and people with HIV, a stigmatized population, often belong to other stigmatized groups as well. This includes MSM, ACB, PWIDs, and CSWs, and is harder to recruit and retain in studies [6–15]. This impacts the ability to study these populations [6–18]. As a result, it is important to employ strategies that reduce barriers to participation and to evaluate them using pilot studies before committing to a full study [33, 55]. Few studies included these key populations, and this could potentially compromise feasibility in the larger study as the challenges in recruiting and retaining them have not been investigated in the pilot.
Our meta-epidemiological study has some weaknesses. Firstly, we were reliant on the authors’ conclusions to determine study feasibility and progression to the larger study. In addition, some of the studies included were evaluated using methodological advances developed after they were published, and this may explain why these approaches (sample size estimation, progression criteria etc.) were not used.
While our study does have weaknesses, it also has several strengths. Our review was robust as our search was highly sensitive, as demonstrated by the exceptionally large number of studies screened (n = 9297). The concepts searched, pilot studies and HIV, were purposefully broad as we are interested in all HIV interventions. In addition, with our study being specific to the HIV literature, we are able to evaluate a particular area of research where pilot studies will increasingly play a bigger role.
Future studies could expand on this work and evaluate the impact pilot studies have on the final study. Furthermore, future studies can assess the change in study quality since the introduction of the CONSORT extension for pilot studies.
Conclusion
Pilot studies are increasingly being used in the HIV field. However, feasibility outcomes were not always the primary outcomes of the pilot studies evaluated. In addition, many key pilot study requirements, such as selecting a sample size, crafting progression criteria, and defining feasibility outcomes were often not included. Higher quality pilot studies are needed.
Supplementary Information
Acknowledgements
None
Abbreviations
- NIHR
National Institute for Health Research
- RfPB
Research for patient benefit
- HIV
Human immunodeficiency virus
- CENTRAL
Cochrane Central Register of Controlled Trials
- WHO
World Health Organization
- RCT
Randomized controlled trial
- ART
Antiretroviral therapy
- ACB
African, Caribbean, and Black
- MSM
Men who have sex with men
- PWID
People who inject drugs
- CSW
Commercial sex workers
- STROBE
STrengthening the Reporting of Observational studies in Epidemiology
- NGT
Nominal group technique
Authors’ contributions
HE and LM designed the study. HE wrote the first draft. HE and LM analyzed the data. HE, MK, SL, OO, ZD, ZM, and AL participated in the title and abstract screening. HE, ZD, OO, and SL participated in the full-text screening. HE and SA extracted the data. HE and LM revised several versions of the manuscript. All authors reviewed the final version of the manuscript. All authors read and approved the final manuscript.
Funding
No funding was used for this study.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Glasziou P. The role of open access in reducing waste in medical research. PLoS Med. 2014;11(5):e1001651. doi: 10.1371/journal.pmed.1001651. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Obstet Gynecol. 2009;114(6):1341–1345. doi: 10.1097/AOG.0b013e3181c3020d. [DOI] [PubMed] [Google Scholar]
- 3.Morgan B, Hejdenberg J, Hinrichs-Krapels S, Armstrong D. Do feasibility studies contribute to, or avoid, waste in research? PLoS One. 2018;13(4):e0195951. doi: 10.1371/journal.pone.0195951. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1. doi: 10.1186/1471-2288-10-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman CL, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS One. 2016;11(3):e0150205. doi: 10.1371/journal.pone.0150205. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Yehia BR, Stewart L, Momplaisir F, Mody A, Holtzman CW, Jacobs LM, et al. Barriers and facilitators to patient retention in HIV care. BMC Infect Dis. 2015;15:246. doi: 10.1186/s12879-015-0990-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Hoffman KA, Baker R, Kunkel LE, Waddell EN, Lum PJ, McCarty D, et al. Barriers and facilitators to recruitment and enrollment of HIV-infected individuals with opioid use disorder in a clinical trial. BMC Health Serv Res. 2019;19(1):862. doi: 10.1186/s12913-019-4721-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Anastasi JK, Capili B, Kim GH, Chung A. Clinical trial recruitment and retention of a vulnerable population: HIV patients with chronic diarrhea. Gastroenterol Nurs. 2005;28(6):463–468. doi: 10.1097/00001610-200511000-00002. [DOI] [PubMed] [Google Scholar]
- 9.De La Rosa M, Babino R, Rosario A, Martinez NV, Aijaz L. Challenges and strategies in recruiting, interviewing, and retaining recent Latino immigrants in substance abuse and HIV epidemiologic studies. Am J Addict. 2012;21(1):11–22. doi: 10.1111/j.1521-0391.2011.00193.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.El-Sadr W, Capps L. The challenge of minority recruitment in clinical trials for AIDS. JAMA. 1992;267(7):954–957. [PubMed] [Google Scholar]
- 11.Fortune T, Wright E, Juzang I, Bull S. Recruitment, enrollment and retention of young black men for HIV prevention research: experiences from the 411 for safe text project. Contemp Clin Trials. 2010;31(2):151–156. doi: 10.1016/j.cct.2009.12.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Silvestre AJ, Hylton JB, Johnson LM, Houston C, Witt M, Jacobson L, et al. Recruiting minority men who have sex with men for HIV research: results from a 4-city campaign. Am J Public Health. 2006;96(6):1020–1027. doi: 10.2105/AJPH.2005.072801. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Cook C, Mack J, Cottler LB. Research participation, trust, and fair compensation among people living with and without HIV in Florida. AIDS Care. 2018;30(1):27–31. doi: 10.1080/09540121.2017.1338656. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Menezes P, Eron JJ, Jr, Leone PA, Adimora AA, Wohl DA, Miller WC. Recruitment of HIV/AIDS treatment-naive patients to clinical trials in the highly active antiretroviral therapy era: influence of gender, sexual orientation and race. HIV Med. 2011;12(3):183–191. doi: 10.1111/j.1468-1293.2010.00867.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Nalubega S, Evans C. Participant views and experiences of participating in HIV research in sub-Saharan Africa: a qualitative systematic review. JBI Database System Rev Implement Rep. 2015;13(5):330–420. doi: 10.11124/jbisrir-2015-2051. [DOI] [PubMed] [Google Scholar]
- 16.Gemmill R, Williams AC, Cooke L, Grant M. Challenges and strategies for recruitment and retention of vulnerable research participants: promoting the benefits of participation. Appl Nurs Res. 2012;25(2):101–107. doi: 10.1016/j.apnr.2010.02.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Batista P, Deren S, Banfield A, Silva E, Cruz M, Garnes P, et al. Challenges in recruiting people who use drugs for HIV-related biomedical research: perspectives from the field. AIDS Patient Care STDs. 2016;30(8):379–384. doi: 10.1089/apc.2016.0135. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Loutfy MR, Logan Kennedy V, Mohammed S, Wu W, Muchenje M, Masinde K, et al. Recruitment of HIV-positive women in research: discussing barriers, facilitators, and research personnel’s knowledge. Open AIDS J. 2014;8:58–65. doi: 10.2174/1874613601408010058. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Asiimwe SB, Kanyesigye M, Bwana B, Okello S, Muyindike W. Predictors of dropout from care among HIV-infected patients initiating antiretroviral therapy at a public sector HIV treatment clinic in sub-Saharan Africa. BMC Infect Dis. 2016;16:43. doi: 10.1186/s12879-016-1392-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bekker LG, Johnson L, Wallace M, Hosek S. Building our youth for the future. J Int AIDS Soc. 2015;18(2 Suppl 1):20027. doi: 10.7448/IAS.18.2.20027. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Macdonald V, Verster A, Baggaley R. A call for differentiated approaches to delivering HIV services to key populations. J Int AIDS Soc. 2017;20(Suppl 4):21658. doi: 10.7448/IAS.20.5.21658. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Djomand G, Quaye S, Sullivan PS. HIV epidemic among key populations in West Africa. Curr Opin HIV AIDS. 2014;9(5):506–513. doi: 10.1097/COH.0000000000000090. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Rao A, Stahlman S, Hargreaves J, Weir S, Edwards J, Rice B, et al. Sampling key populations for HIV surveillance: results from eight cross-sectional studies using respondent-driven sampling and venue-based snowball sampling. JMIR Public Health Surveill. 2017;3(4):e72. doi: 10.2196/publichealth.8116. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Castrillo-Viguera C, Grasso DL, Simpson E, Shefner J, Cudkowicz ME. Clinical significance in the change of decline in ALSFRS-R. Amyotroph Lateral Scler. 2010;11(1-2):178–180. doi: 10.3109/17482960903093710. [DOI] [PubMed] [Google Scholar]
- 25.Kaur N, Figueiredo S, Bouchard V, Moriello C, Mayo N. Where have all the pilot studies gone? A follow-up on 30 years of pilot studies in clinical rehabilitation. Clin Rehabil. 2017;31(9):1238–1248. doi: 10.1177/0269215517692129. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Shanyinde M, Pickering RM, Weatherall M. Questions asked and answered in pilot and feasibility randomized controlled trials. BMC Med Res Methodol. 2011;11:117. doi: 10.1186/1471-2288-11-117. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Chan CL, Leyrat C, Eldridge SM. Quality of reporting of pilot and feasibility cluster randomised trials: a systematic review. BMJ Open. 2017;7(11):e016970. doi: 10.1136/bmjopen-2017-016970. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Mbuagbaw L, Kosa SD, Lawson DO, Stalteri R, Olaiya OR, Alotaibi A, et al. The reporting of progression criteria in protocols of pilot trials designed to assess the feasibility of main trials is insufficient: a meta-epidemiological study. Pilot Feasib Stud. 2019;5:120. doi: 10.1186/s40814-019-0500-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Donovan JL, Rooshenas L, Jepson M, Elliott D, Wade J, Avery K, et al. Optimising recruitment and informed consent in randomised controlled trials: the development and implementation of the quintet recruitment intervention (QRI) Trials. 2016;17(1):283. doi: 10.1186/s13063-016-1391-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Elliott D, Husbands S, Hamdy FC, Holmberg L, Donovan JL. Understanding and improving recruitment to randomised controlled trials: qualitative research approaches. Eur Urol. 2017;72(5):789–798. doi: 10.1016/j.eururo.2017.04.036. [DOI] [PubMed] [Google Scholar]
- 31.UyBico SJ, Pavel S, Gross CP. Recruiting vulnerable populations into research: a systematic review of recruitment interventions. J Gen Intern Med. 2007;22(6):852–863. doi: 10.1007/s11606-007-0126-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Gama A, Martins MO, Dias S. HIV research with men who have sex with men (MSM): advantages and challenges of different methods for most appropriately targeting a key population. AIMS Public Health. 2017;4(3):221–239. doi: 10.3934/publichealth.2017.3.221. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Castillo-Mancilla JR, Cohn SE, Krishnan S, Cespedes M, Floris-Moore M, Schulte G, et al. Minorities remain underrepresented in HIV/AIDS research despite access to clinical trials. HIV Clin Trials. 2014;15(1):14–26. doi: 10.1310/hct1501-14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Schwartz SR, Kavanagh MM, Sugarman J, Solomon SS, Njindam IM, Rebe K, et al. HIV viral load monitoring among key populations in low- and middle-income countries: challenges and opportunities. J Int AIDS Soc. 2017;20 Suppl 7:43–49. [DOI] [PMC free article] [PubMed]
- 35.Jenkins RA. Recruiting substance-using men who have sex with men into HIV prevention research: current status and future directions. AIDS Behav. 2012;16(6):1411–1419. doi: 10.1007/s10461-011-0037-5. [DOI] [PubMed] [Google Scholar]
- 36.Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–142. doi: 10.1136/ebmed-2017-110713. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.The EndNote Team . EndNote. EndNote X7 ed. Philadelphia: Clarivate Analytics; 2013. [Google Scholar]
- 38.Innovation VH. Covidence systematic review software. Melbourne: COVIDENCE; 2014.
- 39.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–381. doi: 10.1016/j.jbi.2008.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Organization WH. Definition of regional groupings 2020. Geneva: WHO; 2020.
- 41.Bank TW. World Bank open data. World Bank. 2018.
- 42.Faber J, Fonseca LM. How sample size influences research outcomes. Dental Press J Orthod. 2014;19(4):27–29. doi: 10.1590/2176-9451.19.4.027-029.ebo. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Baldeh T, MacDonald T, Kosa SD, Lawson DO, Stalteri R, Olaiya OR, et al. More pilot trials could plan to use qualitative data: a meta-epidemiological study. Pilot Feasib Stud. 2020;6(1):164. doi: 10.1186/s40814-020-00712-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Price JT, Mabula-Bwalya CM, Freeman BL, Carda-Auten J, Phiri WM, Chibwe K, et al. Acceptability of a trial of vaginal progesterone for the prevention of preterm birth among HIV-infected women in Lusaka, Zambia: a mixed methods study. PLoS One. 2020;15(9):e0238748. doi: 10.1371/journal.pone.0238748. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. Pilot Feasib Stud. 2016;2:64. doi: 10.1186/s40814-016-0105-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Tsima BM, Moedi P, Maunge J, Machangane K, Kgogwane M, Mudojwa T, et al. Feasibility of implementing a novel behavioural smoking cessation intervention amongst human immunodeficiency virus-infected smokers in a resource-limited setting: a single-arm pilot trial. South Afr J HIV Med. 2020;21(1):1075. doi: 10.4102/sajhivmed.v21i1.1075. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Sackett DL, Cook DJ. Can we learn anything from small trials? Ann N Y Acad Sci. 1993;703:25–31. doi: 10.1111/j.1749-6632.1993.tb26331.x. [DOI] [PubMed] [Google Scholar]
- 48.Cocks K, Torgerson DJ. Sample size calculations for pilot randomized trials: a confidence interval approach. J Clin Epidemiol. 2013;66(2):197–201. doi: 10.1016/j.jclinepi.2012.09.002. [DOI] [PubMed] [Google Scholar]
- 49.Teare MD, Dimairo M, Shephard N, Hayman A, Whitehead A, Walters SJ. Sample size requirements to estimate key design parameters from external pilot randomised controlled trials: a simulation study. Trials. 2014;15:264. doi: 10.1186/1745-6215-15-264. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Whitehead AL, Julious SA, Cooper CL, Campbell MJ. Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable. Stat Methods Med Res. 2016;25(3):1057–1073. doi: 10.1177/0962280215588241. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Viechtbauer W, Smits L, Kotz D, Bude L, Spigt M, Serroyen J, et al. A simple formula for the calculation of sample size in pilot studies. J Clin Epidemiol. 2015;68(11):1375–1379. doi: 10.1016/j.jclinepi.2015.04.014. [DOI] [PubMed] [Google Scholar]
- 52.Hallingberg B, Turley R, Segrott J, Wight D, Craig P, Moore L, et al. Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance. Pilot Feasib Stud. 2018;4:104. doi: 10.1186/s40814-018-0290-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Bertram W, Moore A, Wylde V, Gooberman-Hill R. Optimising recruitment into trials using an internal pilot. Trials. 2019;20(1):207. doi: 10.1186/s13063-019-3296-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.O’Cathain A, Hoddinott P, Lewin S, Thomas KJ, Young B, Adamson J, et al. Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers. Pilot Feasib Stud. 2015;1:32. doi: 10.1186/s40814-015-0026-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Wolf RC, Bingham T, Millett G, Wilcher R. Building the evidence base to optimize the impact of key population programming across the HIV cascade. J Int AIDS Soc. 2018;21 Suppl 5:e25146. doi: 10.1002/jia2.25146. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.