Abstract
Background:
Effective recruitment and retention strategies are essential in clinical trials.
Methods:
The MemAID trial consisted of 12 visits during 24 weeks of intranasal insulin or placebo treatment and 24 weeks of post-treatment follow-up in older people with and without diabetes. Enhanced retention strategies were implemented mid study to address high drop-out rate. Baseline variables used in Cox regression models to identify dropout risk factors were: demographics and social characteristics, functional measures, metabolic and cardiovascular parameters, and medications.
Results:
244 participants were randomized; 13 (5.3%) were discontinued due to adverse events. From the remaining 231 randomized participants, 65 (28.1%) dropped out, and 166 (71.9%) did not. The Non-retention group included 95 participants not exposed to retention strategies, of which 43 (45.2%) dropped out. The Retention group included 136 participants exposed to enhanced retention strategies, of which 22 (16.2%) dropped out. Dropout risk factors included being unmarried, a longer diabetes duration, using oral antidiabetics as compared to not using, worse executive function and chronic pain. After adjusting for exposure to retention strategies, worse baseline executive function composite score (p=0.001) and chronic pain diagnosis (p=0.032) were independently associated with a greater risk of dropping out. The probability of dropping out decreased with longer exposure to retention strategies and the dropout rate per month decreased from 4.1% to 1.8% (p=0.04) on retention strategies.
Conclusions:
Baseline characteristics allow prediction of dropping out from a clinical trial in older participants. Retention strategies has been effective at minimizing the impact of dropout-related risk factors.
Trial Registration:
Clinical trials.gov NCT2415556 3/23/2015 (www.clinicaltrials.gov)
Keywords: clinical trial, dropout risk, Memory Advancement with Intranasal Insulin in Type 2 Diabetes (MemAID) trial, recruitment, retention, type 2 diabetes
INTRODUCTION
Achieving the enrollment goals through effective recruitment and retention is one of the most challenging aspects of conducting clinical trials. A single-center trial design may become arduous for that matter, due to a lower sample size and attrition bias. A multicenter trial design enables to target a larger population at the cost of greater trial complexity, longer duration of the study and higher cost (1). Only 31–55% of randomized controlled trials (RCTs) funded and published by the UK’s National Institute for Health Research Health Technology Assessment Program succeed at achieving the enrollment goals, and up to half of those studies accrue lost to follow-ups of more than 11% (2). Clinician and participant barriers to participation in RCTs often include time constraints, lack of staff and training, additional demands of the trial, participant preferences, worry caused by uncertainty, and concerns about intervention, among others (3). Barriers to recruitment and retention of older people in clinical trials may be related to lack of participant interest, age, and comorbidities, which may be addressed by an engaged research team, flexibility in scheduling, incentives and transportation aids (4).
Discontinuation of participants from a clinical trial can be due to adverse events (AEs), protocol deviations, early termination and dropout (5). Strong efforts to enhance and enforce recruitment and retention strategies must concur with the study design. Attendance to follow-up can be trial and disease specific, and awareness of these features is essential for developing strategies improve retention (6).
Even after successful recruitment and enrollment into the trial, retaining participants remains challenging (7,8). Using numerous strategies spanning different domains may improve retention of participants (9) (e.g. study personnel: enhanced training and management of staff for improved participants contact and scheduling, use of reminders, conducting visits, and adhering to compliance tracking methods; study description: explaining to participants its requirements and details, including potential benefits and risks, as well as reimbursement and other financial and non-financial incentives; creating a study identity for participants: using the study logo, study materials (magnets, mugs, totes), using similar colors and fonts on all study materials; and community involvement). Involuntary discontinuation, such as due to AEs, is difficult to predict, and therefore it is important to implement retention strategies focused on participants at risk of dropping out due to other reasons. For example, studies from weight-loss programs suggest that baseline characteristics can identify participants who are more likely to drop out (10–12).
Identifying baseline characteristics of older participants at risk of dropping out from clinical trials may enable a priori optimization of recruitment and retention strategies to achieve the enrollment goals. There is a lack of reports on dropout patterns in clinical trials of older participants with type 2 diabetes mellitus (T2DM). The Memory Advancement by Intranasal Insulin in Type 2 Diabetes (MemAID) study provided proof-of-concept for preliminary safety and efficacy of the intranasal insulin (INI) for treatment of T2DM and age-related functional decline, as evidenced by a positive effect on verbal learning and executive functioning and faster walking speed, which are important long-term outcomes in older adults (13,14). Enhanced retention strategies were implemented to reduce the dropout rates in this trial.
We aimed to retrospectively identify baseline characteristics predicting the risk of dropping out in the older cohort of the MemAID study. We also assessed the effect of enhanced retention strategies implemented in this RCT on dropout predictors and dropout rate.
METHODOLOGY
Trial design
The MemAID study was a multicenter, prospective, double-blinded, placebo-controlled, parallel design, phase 2 trial, which assessed the long-term effects on cognition and gait of 40 IU human INI (16) once daily compared to placebo (sterile saline) treatment over 24 weeks in elderly participants with and without diabetes with 24 weeks of post-treatment follow-up. Within each site, participants were randomized into the four treatment arms (T2DM-INI or placebo, Control-INI or placebo). The trial was conducted at the Beth Israel Deaconess Medical Center (BIDMC; primary center) and at the Brigham and Women’s Hospital (BWH) in Boston, MA, between 2015 and 2020. Other centers (Joslin Diabetes Clinic and Harvard Medical School) were not involved in participant recruitment. Study screening began on October 6, 2015 at BIDMC and on June 22, 2017 at BWH. In October 2017, enhanced retention strategies were implemented and modifications to the trial protocol were introduced to reduce the dropout rate, following the National Institute of Diabetes and Kidney Diseases (NIDDK) guidance. Due to the COVID-19 pandemic, virtual visits replaced on-site visits since March 25, 2020. The last participant completed a virtual visit on May 21, 2020. The clinical phase was completed on May 31, 2020 and ten participants were censored from the study before the last visit (Visit 12). Further details on study design and methods were reported elsewhere (13–15). The trial was approved by the US Food and Drug Administration (FDA; IND#107690), registered on www.clinicaltrials.gov (NCT02415556), and conducted in accordance with the Guidelines for Good Clinical Practice and followed Consolidated Standards for Reporting in Clinical Trials (CONSORT).
Study protocol
The study protocol was approved by the BIDMC Institutional Review Board and all volunteers signed the informed consent form. Participants were recruited from advertisements in local newspapers, radio, BIDMC, BWH and Joslin Diabetes Center clinics, ClinQ BIDMC and Research Match databases. Enrollment involved a telephone pre-screening and an on-site screening (Visit 1) followed by randomization. Baseline and first intervention visits occurred on the same day after randomization (Visit 2 baseline and Visit 2 intervention). At baseline, a comprehensive past medical history was obtained from participants and health records. The Charlson Co-morbidity Index (CCI) score was calculated using ICD-10 codes relating to reported comorbidities. After the first intervention (Visit 2, day 0), the 24-week treatment period consisted of three assessment visits at eight weeks intervals (Visits 4, 6, and 8) intercalated with three medication refills (Visits 3, 5, 7). Visit 8 was the end-of-treatment. The 24-week post-treatment period consisted of one telephone interview (Visit 9) one week after end-of-treatment and three assessment visits (Visit 10–12), each at eight weeks intervals. Assessment visits included fasting metabolic panels, anthropometrics, cognitive, functional and mood tests, and normal and dual-task walks. Participants documented daily medication usage and self-monitored blood glucose weekly. Adverse events and medication adherence were monitored at every study visit. Study physicians evaluated participants at baseline and at any adverse event occurrence.
Inclusion and exclusion criteria
Inclusion criteria were men and women 50–85 years old and able to walk for six minutes. T2DM participants were being treated with either diet, non-insulin oral or injectable antidiabetic drugs. Controls had normal fasting plasma glucose (FPG; <126 mg/dL) and glycated hemoglobin (HbA1c; <6.5%). Exclusion criteria were type 1 diabetes mellitus, history of severe hypoglycemia or a clinically acute medical condition that required either hospitalization or surgery within the past six months, liver or renal failure or transplant, diagnosis of dementia or Mini-Mental State Examination (MMSE) score ≤20, current recreational drug or alcohol abuse or serious systemic disease. We excluded participants with more than one hypoglycemic episode during the study. After implementation of enhanced retention strategies in 2017, enrollment of T2DM participants treated with systemic insulins was stopped.
Intervention and randomization
Participants self-administered 40 IU (0.4 mL) of human insulin (rDNA; Novolin® R NovoNordisk Inc., Baksvaerd, Denmark, off label use) (16) or placebo (0.4 mL bacteriostatic sodium chloride 0.9% solution) intranasally once daily before breakfast for 24 weeks. The ViaNase device (Kurve technologies Inc., Lynnwood, WA) was used to deliver medication. Randomization into four groups used three block sizes (four, eight, and twelve) to achieve uniform distribution and was designed by the study statistician (L.N.). BIDMC and BWH pharmacies reconstituted and distributed medication. The principal investigator, study physicians, staff, participants, and their health care providers were blinded.
MemAID trial main outcomes
The MemAID study primary outcomes were cognition (memory and executive function), normal and dual-task gait speeds. Cognitive measures were evaluated using the Cambridge Neuropsychological Test Automated Battery CANTAB (Cambridge Cognition, Ltd, Bottisham, UK) (17). CANTAB tests includes precise and objective measures of cognitive function that have demonstrated sensitivity to detecting changes in neuropsychological performance and include tests of working memory, learning and executive function; visual, verbal and episodic memory; attention, information processing and reaction time; social and emotion recognition, decision making and response control. Cognitive and executive function variables were converted to scaled z-scores, and summed to create composite scores (18). The executive function composite score consisted of paired associates learning (PAL, total errors; lower score better) and spatial working memory (SWM, total errors and strategy to complete tasks; lower score better). The verbal memory composite score consisted of verbal immediate free recall, immediate and delayed verbal recognition memory (VRM; higher score better).
Gait speed was measured using the Mobility Lab System (APDM, Inc., Portland, OR) over six minutes hallway walking, excluding turns. Dual task walking test involved counting backwards subtracting by seven while walking for six minutes.
Dropouts
Dropouts were participants who were discontinued from the study after randomization due to reasons different from AEs. “Non-dropouts” were participants who completed the study, either at Visit 12 or earlier due to clinical phase completion. Reasons for dropping out were investigated by the study team before discontinuation and categorized as participant-related issues, family or health-related issues, and lost to follow-up.
Retention strategies
The original study design required 12 on-site visits. Incentives to participation were remuneration for study completion up to 1,000 USD, meals and reminders. Enhanced retention strategies were implemented in November 2017, due to a high dropout rate. Enrollment of T2DM participants treated with systemic insulins, who showed a 65% dropout rate, was stopped by the Data and Safety Monitoring Board following NIDDK guidance in October 2017 (15).
Retention strategies included: change of eligibility criteria (exclusion of T2DM participants treated with systemic insulins and no upper limit on body mass index (BMI); re-screening participants who were ineligible before but became eligible after changing eligibility criteria, change in remuneration schedule (a higher payment for visits during the treatment period including skipped visits or phone visits; although some participants skipped visits during the treatment period, these were partially remunerated after demonstrating commitment to the protocol by showing daily administration of study medication, completing study diaries and questionnaires), reminder letters and phone calls prior to each visit, flexibility in scheduling (interim visits, skipping visits, phone visits, mailing medications), incentives (mugs, totes, and pens), transportation aids (Uber, Lyft, taxi, public transportation, gas reimbursement), and 24/7 contact with investigators. BMI was initially set as an exclusion criterion because the magnetic resonance imaging suite would not fit participants over the weight limit. After the suite was upgraded, this exclusion criterion was no longer needed and head circumference was used.
Participants were categorized into “Retention group” and “Non-retention group”. The Retention group included participants who were randomized after November 2017 or were active in the study when enhanced retention strategies were implemented, that is, those who received these benefits. The Non-retention group included participants who completed the study or were discontinued before November 2017, and thus did not receive the benefits. Days of exposure to retention strategies were calculated for each participant during their active course in the study.
Statistical analysis
Data were converted from MemAID database (Study TRAX© (Macon, GA, USA). The code and data analyses were generated using Statistical Analyses Software (SAS), Version 9.4 TS level; SAS System for Windows (X64_8PRO platform, Copyright © 2002–2012 SAS Institute Inc. (Cary, NC, USA) and JMP® Pro, Version 15 (SAS Institute Inc. Cary, NC, USA).
Dropout predictors and the effect of retention strategies on the risk of dropping out
Baseline characteristics were screened for potential predictors of dropping out by identifying significant differences between the Dropout and Non-dropout groups using the Fisher’s exact test and the Wilcoxon rank sums test. The variables included: demographics (age, sex, race, ethnicity, marital status, employment status, years of education, and advertisement source of enrollment), functional measures (normal and dual task gait speed, executive function composite z score, verbal memory composite z score, the Wechsler adult reading test (adjusted scale 50–128; WTAR-IQ), the Mini-Mental State Examination (MMSE), the CCI score, the World Health Organization Disability Assessment Schedule (WHODAS) 2.0, Geriatric Depression Scale score, chronic pain diagnosis, and Toronto neuropathy scale score), metabolic and cardiovascular (diabetes diagnosis and duration, BMI, waist circumference, HbA1c, fasting glucose, homeostatic model of insulin resistance (HOMA-IR), total cholesterol and hypertension) and medications. The advertisement source of enrollment was classified as “out-of-hospital” (internet advertisements, newspaper/magazine advertisements, referrals, TV/radio advertisements) as compared to “in-hospital” (clinic referrals, in-hospital flyers, clinical databases). The WTAR-IQ measures pre-morbid intellectual functioning in individuals aged 16–89 and can be used to predict IQ. Logistic regression analysis tested the unadjusted effect of being in either the Retention or the Non-retention groups on the probability of dropout.
We used purposeful selection to identify a multivariate model of risk factors for dropping out. As a time-to-event variable, the number of days being active in the study was calculated for each participant by subtracting the date of randomization from the day of termination or discontinuation (time-to-completion vs. time-to-dropout). Censored participants included those discontinued due to adverse events and those who completed the study before Visit 12 due to clinical phase closure. The proposed multivariate model was analyzed by means of a Cox regression model to quantify the dropout hazards for each dropout risk factor. The exposure to retention strategies was used as a continuous variable into the Cox regression model to test its effect on the dropout hazards (19). Kaplan-Meier curves were plotted to calculate the probabilities of dropping out throughout the study and adjusted for the Non-retention and Retention groups, and differences were tested by means of the log-rank test.
The effect of retention strategies on dropout rates
A monthly dropout rate throughout the study was calculated as a proportion of dropouts relative to the number of active participants in each month between the beginning and the end of the trial (October 2015-May 2020). There were 24 months before retention strategies (October 2015–2017), and 30 months on retention strategies (November 2017-May 2020). Dropout rate per month, before and after implementation of enhanced retention strategies, was compared by two-sided t-test.
RESULTS
Participant flow
Figure 1 summarizes participants’ enrollment in the MemAID study. From the 668 screened participants, 244 were randomized and 13 (5.3%) were discontinued due to AEs. From the resting 231 randomized participants, 65 (28.1%) dropped out, and 166 (71.9%) did not. From these 166 participants, 78 (47%) were diabetic and 88 (53%) non-diabetic participants; 156 completed the full length of the study and 10 were censored due to clinical phase closure. From the 65 dropouts, 30 (46.2%) were diabetic and 35 (53.8%) non-diabetic participants. Systemic insulin-treated T2DM participants had the highest dropout rate (65%).
Figure 1.
CONSORT diagram for MemAID trial by the Non-retention and Retention groups.
The Non-retention group included 95 participants (66.3 ± 9.5 years old; 46 [48%] female), of which 43 (45.2%) dropped out. From these 43 dropouts, 16 (37.2%) occurred in the pre-treatment, 23 (53.5%) in the treatment, and 4 (9.3%) in the post-treatment periods. The Retention group included 136 participants (64.6 ± 8.3 years old; 67 [49%] female) who were active when enhanced retention strategies were implemented (42), or randomized afterwards (94), of which 22 (16.2%) dropped out. From these 22 dropouts, 6 (27.3%) occurred in the pre-treatment, 13 (59.1%) in the treatment, and 3 (13.6%) in the post-treatment periods.
The median follow-up for the whole cohort was 358 days (interquartile range (IQR): 133–358), whereas 63 days (IQR: 33–134) for the Dropout group, 358 (IQR: 358–358) days for the Non-dropout group, 358 days (IQR: 330–358) for the Retention group, and 358 days (IQR: 81–358) for the Non-retention group.
Reasons for dropping out
From the 65 dropouts, 32 (49.2%) reported participant-related issues, 17 (26.2%) family or health-related issues, and 16 (24.6%) were lost to follow-up. From the 43 dropouts among the Non-retention group, 18 (41.9%) reported participant-related issues, 15 (34.9%) family or health-related issues, and 10 (23.3%) were lost to follow-up. From the 22 dropouts among the Retention group, 14 (63.6%) reported participant-related issues, 2 (9.1%) family or health-related issues, and 6 (27.3%) were lost to follow-up. Reported participant-related issues were mainly related to either lack of interest in participating or not willing to commit to the protocol. These were reported by 16 participants in the Non-retention group (88.9%) and 11 in the Retention group (78.6%).
Baseline characteristics of dropouts
Table 1 summarizes baseline characteristics of the 231 randomized participants, classified as Dropouts and Non-dropouts. Dropout participants were more likely to be unmarried (75% vs. 55%, p=0.004), enrolled from an out of hospital advertisement source (82% vs. 67%, p=0.035), had worse executive function performance (composite z score 1.8 vs. 0.4, p<0.0001), fewer years of education (15.3 vs. 16.4, p=0.026), higher prevalence of chronic pain (34% vs. 21%, p=0.041), and insulin-treated T2DM (12% vs. 3%, p=0.010). Employment status unemployed was borderline (25% vs. 12%, p=0.052).
Table 1.
Baseline characteristics.
| Whole Cohort | Dropouts | Non-dropouts | p-value | |
|---|---|---|---|---|
| Number of participants | 231 | 65 | 166 | |
| Demographics | ||||
| Age, years | 65.3 ± 8.9 | 65.3 ± 9.1 | 65.3 ± 8.8 | 0.886 |
| Women, n (%) | 113 (49) | 34 (52) | 79 (48) | 0.560 |
| Race, n (%) | 0.556 | |||
| White, n (%) | 182 (79) | 52 (80) | 130 (78) | |
| Black, n (%) | 33 (14) | 9 (14) | 24 (15) | |
| Asian, n (%) | 9 (4) | 1 (2) | 8 (5) | |
| Other, n (%) | 7 (3) | 3 (5) | 4 (2) | |
| Ethnicity - Hispanic, n (%) | 13 (6) | 6 (9) | 7 (4) | 0.200 |
| Unmarried, n (%) | 140 (61) | 49 (75) | 91 (55) | 0.004 |
| Employment status, n (%) * | 0.052 | |||
| Employed | 82 (36) | 20 (31) | 62 (38) | |
| Unemployed | 35 (15) | 16 (25) | 19 (12) | |
| Retired | 114 (49) | 29 (45) | 85 (51) | |
| Education, years | 16.1 ± 3.4 | 15.3 ± 3.4 | 16.4 ± 3.4 | 0.026 |
| Out of hospital source of enrollment, n (%) | 164 (71) | 53 (82) | 111 (67) | 0.035 |
| Functional measures | ||||
| Gait speed, cm/s | 115.1 ± 21.6 | 113.1 ± 21.1 | 115.6 ± 21.7 | 0.618 |
| Dual-task gait speed, cm/s | 104.9 ± 22.9 | 102.5 ± 22.7 | 105.6 ± 23.0 | 0.566 |
| Executive function, composite z score | 0.7 ± 2.2 | 1.8 ± 1.9 | 0.4 ± 2.2 | <0.001 |
| Verbal memory, composite z score | −0.4 ± 2.7 | −0.8 ± 2.7 | −0.3 ± 2.6 | 0.208 |
| IQ WTAR adjusted, (range 50–128) | 112.3 ± 13.9 | 112.0 ± 13.8 | 112.4 ± 14.0 | 0.994 |
| Mini-Mental State Exam, (range 0–30) | 28.3 ± 1.8 | 28.1 ± 2.0 | 28.4 ± 1.7 | 0.618 |
| Charlson Co-morbidity Index, (range 0–24) | 3.3 ± 1.7 | 3.4 ± 1.8 | 3.3 ± 1.7 | 0.930 |
| WHODAS 2.0 complex, (range 0–100) | 11.9 ± 12.2 | 13.5 ± 12.3 | 11.5 ± 12.2 | 0.366 |
| Geriatric Depression Scale, (range 0–30) | 5.6 ± 5.2 | 6.4 ± 5.3 | 5.4 ± 5.3 | 0.234 |
| Toronto Neuropathy Scale, (range 0–19) | 3.1 ± 3.5 | 2.8 ± 3.5 | 3.2 ± 3.5 | 0.223 |
| Chronic pain diagnosis, n (%) | 56 (24) | 22 (34) | 34 (21) | 0.041 |
| Metabolic and cardiovascular | ||||
| T2DM diagnosis, n (%) | 108 (47) | 30 (46) | 78 (47) | 1.000 |
| T2DM duration, years | 5.4 ± 7.9 | 7.0 ± 10.2 | 4.7 ± 6.7 | 0.462 |
| BMI, Kg/m2 | 29.5 ± 6.1 | 29.1 ± 5.5 | 29.7 ± 6.3 | 0.618 |
| Waist circumference, cm | 104.0 ± 16.1 | 103.1 ± 16.2 | 104.4 ± 16.1 | 0.457 |
| Hemoglobin A1c, % | 6.4 ± 1.3 | 6.6 ± 1.7 | 6.3 ± 1.1 | 0.520 |
| Fasting plasma glucose, mg/L | 114.4 ± 43.9 | 121.8 ± 54.2 | 111.4 ± 39.0 | 0.224 |
| HOMA-IR | 3.2 ± 3.3 | 3.7 ± 3.4 | 3.1 ± 3.2 | 0.107 |
| Total cholesterol, mg/dL | 178.9 ± 42.7 | 183.4 ± 40.2 | 177.1 ± 43.6 | 0.218 |
| Hypertension diagnosis, n (%) | 109 (47) | 32 (49) | 77 (46) | 0.770 |
| Medications | ||||
| Use of subcutaneous insulin, n (%) | 13 (6) | 8 (12) | 5 (3) | 0.010 |
| Use of oral antidiabetic drugs, n (%) | 96 (42) | 23 (35) | 73 (44) | 0.299 |
| Use of injectable antidiabetic drugs, n (%) | 9 (4) | 3 (5) | 6 (4) | 0.714 |
| Use of antihypertensive drugs, n (%) | 114 (49) | 34 (52) | 80 (48) | 0.661 |
| Use of lipid lowering drugs, n (%) | 112 (48) | 28 (43) | 84 (51) | 0.310 |
IQ-WTAR: Intelligence Quotient - Wechsler Test of Adult Reading; WHODAS: WHO Disability Assessment Schedule; T2DM: Type 2 Diabetes Mellitus; BMI: Body Mass Index; HOMA-IR: Homeostatic Model Assessment for Insulin Resistance.
Data are mean ± SD unless otherwise indicated.
Wilcoxon and Fisher’s exact test tested comparisons between dropouts and non-dropouts.
Bold variables denote p-values <0.05.
Baseline characteristics of the 231 randomized participants were compared between Retention and Non-retention groups. Participants in the Retention group were more likely to have higher systolic blood pressure (16.2% vs. 15.4%, p=0.030), and less likely to be unmarried (54.4% vs. 69.5%, p=0.028) and to use systemic insulins (1.5% vs. 11.6%, p=0.002) or NSAIDs (8.1% vs. 17.9%; p=0.039). White race, elevated systolic blood pressure, and use of oral antidiabetics were borderline (respectively: 74.3% vs. 85.3%, p=0.050; 35.3% vs. 24.2%, p=0.083; and 47.1% vs. 33.7%, p=0.057).
Table 2 summarizes the univariate survival analysis of baseline characteristics, suggesting that risk factors (HR and 95% CI) for dropping out are being unmarried (HR 2.13, p=0.009), unemployed as compared to employed (HR 2.13, p=0.024) and retired (HR 2.11, p=0.017), enrolled from an out of hospital advertisement source (HR 1.94,p=0.039), having worse executive function (HR 1.28, p=<0.001), and history of chronic pain (HR 1.72,p=0.039). Having more years of schooling (HR 0.92, p=0.023) and not using systemic insulin (HR 0.37, p=0.009) seems to be protective against the risk of dropout. As per the purposeful selection method we identified that major risk factors for dropping out from the MemAID study were: being unmarried, longer T2DM duration, not using oral antidiabetics, worse executive functioning and chronic pain. For this multivariate model, predictive performance was estimated by a C-Statistic of 79.2%.
Table 2.
Survival analysis of dropout predictors. Hazard ratios were calculated using a Cox regression method.
| Univariate analysis of Baseline Characteristics | Dropout Hazard Ratio | 95% Confidence Interval | p-value | Multivariate analysis of Baseline Characteristics | Dropout Hazard Ratio | 95% Confidence Interval | p-value |
|---|---|---|---|---|---|---|---|
| Age, per year | 0.98 | 0.97 – 1.03 | Executive function composite z score, per 1.0 units | 1.27 | 1.09 – 1.48 | 0.002 | |
| Women, yes vs no | 1.15 | 0.71 – 1.88 | Chronic pain diagnosis, yes vs no | 1.94 | 1.02 – 3.69 | 0.044 | |
| White, yes vs no | 1.01 | 0.60 – 2.02 | Use of oral antidiabetic drugs, no vs. yes | 3.35 | 1.55 – 7.26 | 0,002 | |
| Black, yes vs no | 0.94 | 0.47 – 1.91 | T2DM duration, per year | 1.05 | 1.01 – 1.09 | 0,007 | |
| Asian, yes vs no | 0.38 | 0.05 – 2.74 | Unmarried, yes vs no | 2.46 | 1.21 – 5.05 | 0.013 | |
| Other, yes vs no | 1.53 | 0.48 – 4.88 | |||||
| Ethnicity - Hispanic, yes vs no | 1.71 | 0.74 – 3.95 | |||||
| Unmarried, yes vs no | 2.13 | 1.21 – 3.75 | 0.009 | ||||
| Unemployed as compared to employed | 2.13 | 1.10 – 4.12 | 0.024 | ||||
| Unemployed as compared to retired | 2.11 | 1.14 – 3.88 | 0.017 | ||||
| Out of hospital source of enrollment, yes vs no | 1.94 | 1.04 – 3.63 | 0.039 | ||||
| Gait speed, per cm/s | 1.00 | 0.98 – 1.01 | |||||
| Dual-task gait speed, per cm/s | 0.99 | 0.98 – 1.01 | |||||
| Executive function composite z score, per 1.0 unit | 1.28 | 1.11 – 1.47 | <0.001 | ||||
| Verbal memory composite z score, per 1.0 unit | 0.96 | 0.86 – 1.05 | |||||
| Education, per year | 0.92 | 0.85 – 0.99 | 0.023 | ||||
| IQ WTAR adjusted, per 1 unit | 1.00 | 0.98 – 1.02 | |||||
| Mini-Mental State Exam, per 1 unit | 0.96 | 0.84 – 1.10 | |||||
| Charlson Co-morbidity Index, per 1 unit | 1.00 | 0.86 – 1.16 | |||||
| WHODAS 2.0 complex, per 1 unit | 1.01 | 0.99 – 1.03 | |||||
| Geriatric Depression Scale, per 1 unit | 1.03 | 0.98 – 1.08 | |||||
| Toronto Neuropathy Scale, per 1 unit | 0.98 | 0.90 – 1.04 | |||||
| Chronic pain diagnosis, yes vs no | 1.72 | 1.03 – 2.87 | 0.039 | ||||
| T2DM diagnosis, yes vs no* | 0.95 | 0.56 – 1.55 | |||||
| T2DM duration, per year | 1.02 | 1.00 – 1.05 | 0.088 | ||||
| BMI, per 1 Kg/m2 | 0.99 | 0.94 – 1.01 | |||||
| Waist circumference, per cm | 1.00 | 0.98 – 1.01 | |||||
| Hemoglobin A1c, per 1%* | 1.17 | 0.98 – 1.37 | 0.080 | ||||
| Fasting plasma glucose, per mg/L | 1.00 | 1.00 – 1.01 | |||||
| HOMA-IR, per 1 unit | 1.04 | 0.97 – 1.10 | |||||
| Total cholesterol, per mg/dL | 1.00 | 1.00 – 1.01 | |||||
| Hypertension diagnosis, yes vs no | 1.07 | 0.66 – 1.75 | |||||
| Use of systemic insulin, no vs. yes | 0.37 | 0.18 – 0.78 | 0.009 | ||||
| Use of oral antidiabetic drugs, no vs. yes | 1.39 | 0.84 – 2.31 | |||||
| Use of injectable antidiabetic drugs, no vs. yes | 0.80 | 0.25 – 2.56 | |||||
| Use of antihypertensive drugs, no vs. yes | 0.88 | 0.54 – 1.44 | |||||
| Use of lipid lowering drugs, no vs. yes | 1.24 | 0.76 – 2.03 |
IQ-WTAR: Intelligence Quotient - Wechsler Test of Adult Reading; WHODAS: WHO Disability Assessment Schedule; T2DM: Type 2 Diabetes Mellitus; BMI: Body Mass Index; HOMA-IR: Homeostatic Model Assessment for Insulin Resistance.
The Wald’s test was used for comparisons between dropouts and non-dropouts.
Bold variables denote p-values <0.05, p-values >0/.1 are not listed.
Borderline p-values.
C-Statistic for multivariate model: 0.792.
Retention strategies
Retention strategies significantly reduced the risk of dropping out. Before implementing enhanced retention strategies, 43 out of 95 (45.3%) randomized participants dropped out. After implementing enhanced retention strategies, 22 out of 136 (16.2%) randomized participants dropped out (Figure 1). The probability of non-dropping out was 47.9% after one day of exposure to retention strategies, steeply increasing with more cumulative days of exposure to ~71% at 90 days, and being 97.5% near the end of the post-treatment follow-up (Figure 2). The monthly dropout rate declined from 4.1% to 1.9% after implementing enhanced retention strategies (p=0.049).
Figure 2:
The probability of non-dropping out/retention was calculated by univariate Cox regression for participants active in the study as a function of number of days of exposure to retention strategies. Probability of dropping out declined non-linearly with longer duration of retention strategies. The probability of non-dropping out increased from 47.9% (one day on retention strategies; 52.1 % probability of dropping out) to 71% after ~90 days on retention strategies, and continued to increase throughout the study to 97.5% at the end of follow-up.
Figure 3 depicts Kaplan-Meier curves for dropouts for the Non-retention and Retention groups. The Non-retention group had overall higher probability of dropping out (45 % vs. 17%, p<0.0001). For both groups, the risk for dropping out was the highest at the beginning of study (up to ~100 days), and was 33% in the Non-retention group compared to 12% in the Retention group. In the Retention group, dropouts become sporadic by the second half of 168 days treatment period till the end of study.
Figure 3:
The Non-retention group had overall higher probability of dropping out (45% vs. 17%, p<0.0001). The risk for dropping out was the highest at the beginning of the study (up to ~100 days), and was 33% in the Non-retention group compared to 12% in the Retention group. In the Retention group, dropouts became sporadic after the second half of the 168 days treatment period till the end of study. P value denotes log-rank test comparisons.
Exposure to retention strategies decreased dropout hazard of per one month vs. non-exposure. Table 3 shows the unadjusted effect of univariate analysis of exposure to retention strategies, and a multivariate model of exposure to retention strategies adjusted for baseline predictors of dropping out. Exposure to retention strategies strongly reduced the risk of dropping out in univariate and multivariate model (HR 0.76, p<0.001 vs. 0.80, p<0.001). According to univariate model (Table 3), each month of exposure to retention strategies was associated with a dropout hazard reduction of 20% (HR 0.80, 95% CI [0.72–0.87], p<0.001). The effects of baseline predictors on the risk of dropping out were modified by the exposure to retention strategies vs. non-exposure. For this multivariate model, predictive performance was estimated by a C-Statistic of 88.4%. The effects of executive function composite score (HR 1.29, p=0.001) and chronic pain (HR 2.07, p=0.032) remained strong, but the effect of oral antidiabetics (HR 2.25, p=0.051) was borderline, and T2DM duration (HR 1.04, p=0.066) and unmarried status (HR 1.54, p=0.248) were not significant.
Table 3.
Drop out hazard per month of exposure to retention strategies (unadjusted) and adjusted for baseline predictors of dropping out.
| Risk Factors | Unadjusted effect of retention strategies | Adjusted effect of retention strategies | ||
|---|---|---|---|---|
| Hazard Ratio (95% CI) | p-value | Hazard Ratio (95% CI) | p-value | |
| Retention strategies, per month | 0.76 (0.69 – 0.83) | <0.001 | 0.80 (0.72 – 0.87) | <0.001 |
| Executive function composite z score, per 1.0 units | - | - | 1.29 (1.11 – 1.50) | 0.001 |
| Chronic pain diagnosis, yes vs. no | - | - | 2.07 (1.07 – 4.00) | 0.032 |
| Use of oral antidiabetic drugs, no vs. yes | - | - | 2.25 (1.00 – 5.08) | 0.051 |
| T2DM duration, per year | - | - | 1.04 (1.00 – 1.08) | 0.066 |
| Unmarried, yes vs. no | - | - | 1.54 (0.74 – 3.20) | 0.248 |
Dropout hazard per month
The univariate analyses of an unadjusted effect of exposure to retention strategies (per month) in dropouts vs. non-dropouts.
The multivariate analysis of exposure to retention strategies adjusted for baseline predictors of dropouts. Baseline predictors of drop-out were identified by a purposeful selection method.
Hazard ratios were calculated using a Cox regression method.
The Wald’s test was used for comparisons between dropouts and non-dropouts.
C-Statistic for multivariate model: 0.884.
DISCUSSION
The MemAID study was an RCT of INI or placebo treatment in older diabetic and non-diabetic participants over 24 weeks with 24 weeks post-treatment follow-up. The MemAID study provided evidence for preliminary safety and efficacy of INI for treatment of T2DM and age-related cognitive and mobility decline, and determined that INI is safe and does not cause hypoglycemic episodes, or interferes with subcutaneous insulin treatments (13,15). This retrospective quasi experimental study is a sub-analysis of the MemAID study. We identified baseline characteristics predicting the risk of dropping out and demonstrated the effect of retention strategies on dropout predictors and dropout rate.
The overall dropout rate was 28.1% (65 out of 231 randomized participants). Enhanced retention strategies (changing eligibility criteria, exclusion of systemic insulin-treated T2DM volunteers, flexibility in visit scheduling, incentives and transportation aids) were implemented mid-study. Exposure to retention strategies reduced the overall dropout rate from 45.2% to 16.2% and monthly drop out rate from 4.1% to 1.9%, and attenuated the effects of baseline predictors of dropping out.
Enrollment goals were achieved at 100% in control and at 78% in diabetic groups (13). T2DM participants treated with systemic insulins were the most challenging group to recruit and retain. They had a high rate of screen failures and 65% dropout rate due to multiple comorbidities meeting exclusion criteria, lack of interest in participation, and hypoglycemic episodes (15). Although the use of subcutaneous insulin appears as a strong predictor of dropout in the univariate analysis (p=0.010), its effect was nonsignificant when included in the multivariate models exposed in Table 3 (without retention: p=0.156; with retention: p=0.458). This may be explained by a relatively low number of observations (13 participants within the whole cohort were on insulin).
Worse executive function and chronic pain were major risk factors for dropping out despite exposure to retention strategies. Executive function is the cognitive domain that comprises complex task execution, planning and decision-making. Our composite score specifically measured the ability to store and recall short-term information (PAL) and the ability to retain and manipulate visuospatial information (SWM). Poor executive functioning may pose participants at risk of dropping out by compromising their ability to commit with the study protocol and its complexity. Self-reported non-adherence to diabetes treatment among older Hispanic adults correlates with worse performance in the executive clock drawing task and worse HbA1c status (17). Chronic pain can affect participant’s willingness to comply with the complexity of study protocol as it can be associated with limited mobility, worse mood and a higher risk of falling and lower health-related quality of life in the elderly (20).
Barriers to recruitment and retention of older people in clinical trials could be addressed by an engaged research team, flexibility in scheduling, incentives, and transportation aids (4). Our retention strategies were tailored to the study population by targeting multiple domains and decreased monthly drop out rate from 4.1% to 1.9%, which is consistent with other interventions (6). Our time-to-event analysis demonstrated that the initial separation of probability of dropout between groups occurred around the time of the first on-site assessment visit. By then, MemAID participants have received welcome on study letters, bags, brochures and other materials as well as reminder letters, phone calls and transportation benefits.
Dropouts were more likely to occur earlier in the treatment period, about three months after randomization, which may reflect a lack of interest in participating or not willing to commit to the protocol. Therefore, investigators need to be aware of red flags suggesting a participant is at higher risk of dropping out. However, they should also be aware of the disparity between minimizing attrition bias by excluding participants with higher risk of dropping out and maximizing external validity. Further research is needed for better understanding of dropout risk predictors in the setting of other RCTs (e.g., cohort demographics and clinical characteristics, treatment demands, trial duration, etc.). Continuous monitoring of dropout patterns and reasons for dropping should be carried out during clinical phases of RCTs to early identify potential predictors that are trial-specific. Such predictors could be used to tailor enhanced and effective retention strategies as demonstrated by this study.
Successful retention requires taking into consideration participant’s characteristics, such as education level, reasons and motivations for participating, and barriers to participation like concerns regarding treatment arms or time constraints (1). Successful trials achieving more than 80% retention of participants (which recruited more than 200 subjects who were followed-up for at least one year), often had well-trained study staff and strategies in place that were dynamic and tailored to participants’ characteristics (21). The successful strategies that are often implemented to enhance retention include contact and scheduling methods, visit characteristics, study personnel, financial and non-financial incentives, reminders, special tracking methods, study description, benefits of study, reimbursement, study identity, and community involvement (9,21). Our strategies were consistent with these recommendations and effectively reduced the dropout rates. These were tailored to the population characteristics, aiming to reduce participant’s burden by the length and complexity of the study protocol. Flexibility in scheduling allowed us to accommodate our participant’s schedules, especially for those who traveled long distances to the study site. Phone call reminders prior to each visit improved attendance to visits. The new payment structure offered higher financial incentives for participants throughout the study, receiving the higher amounts for assessment visits and medication refill visits at the beginning of treatment when the likelihood of dropping out was higher. Participants were also compensated for skipped visits during treatment and follow-up, as long as they completed medication usage logs and returned study materials and questionnaires. Compensation for skipped visits encouraged participants to continue treatment and to complete the follow-up. Reimbursements of travel costs removed transportation barriers and participants often used and appreciated scheduled rides.
Potential limitation was the inability to identify which of the individual components of the enhanced retention strategies was in fact the most effective, or if any was ineffective, because all strategies were implemented at the same time. However, retention strategies as a whole demonstrated to be effective. The study design required a high level of commitment from participants given the complexity of the protocol and one-year duration, which posed challenges the older participants with multiple health problems. Furthermore, the COVID-19 pandemic reduced recruitment and enrollment as it may have affected the willingness of older volunteers to be involved in a hospital-based clinical trial.
CONCLUSIONS
Implementation of enhanced retention strategies reduced dropout probability in the MemAID trial. Worse executive function and chronic pain diagnosis at baseline remained independent predictors of dropping out. Retention strategies attenuated the effects of social and diabetes-related factors. Identifying baseline characteristics of participants at risk of dropping out in combination with strategies tailored to older population may improve enrollment and retention in geriatric studies.
ACKNOWLEDGEMENTS
The authors thank to MemAID Investigators, BIDMC and BWH CRC nurses and staff for their contributions, time and skills for completion of this study.
Research reported in this publication was supported by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health (NIDDK) under Award Number R01DK103902 to Vera Novak (FDA IND 107690, clinical trials.gov registration NCT2415556), and with support from Harvard Catalyst - The Harvard Clinical and Translational Science Center (National Center for Advancing Translational Sciences, National Institutes of Health Award UL 1TR002541) and financial contributions from Harvard University and its affiliated academic healthcare centers. The content is solely the responsibility of the authors and does not necessarily represent the official views of Harvard Catalyst, Harvard University, and its affiliated academic healthcare centers, or the National Institutes of Health. The research in this study was supported with study drug from NovoNordisk Inc.; Bagsværd, Denmark through an independent ISS grant (ISS-001063) (to V.N). Safety sub-study was supported with CGM monitoring devices and supplies from Medtronic Inc., Northridge CA, USA through an independent grant NERP15-031 to V.N.
DISCLOSURES
V. Novak, F. Khan and D. Isaza-Pierotti report no disclosures relevant to the manuscript.
C.S. Mantzoros has served as consultant for Novo Nordisk, Inc. Bagsværd, Denmark on Obesity Advisory Board, and has received grant support through BIDMC, which could be considered as related to this project given that Novo Nordisk, Inc. provided medication. C.S. Mantzoros has served as consultant for Coherus, Redwood City CA, AltrixBio Cambridge, MA, California Walnut Commission, Folsom, CA, Genfit, Cambridge MA, Regeneron, Wechester NY, Ansh, Webster TX, Amgen, Thousand Oaks CA, Intercept, New York, NY, 89Bio, has received grant support from Coherus, Redwood City CA, AltrixBio Cambridge, MA, Merck, Rahway City, NJ and has provided educational services through Elsevier, New York, NY, CMHC, Baton Rouge, FL, TMIOA, Tarzana, CA (all unrelated to this project since 2020–2022).
L. Ngo provided consultation to the Radiological Society; to the Journal of Cardiovascular Magnetic Resonance; to Five Island Consulting LLC, Georgetown ME; and to Vinmec Inc. Hanoi, Vietnam between 2015 and 2020.
P. Novak is advisor - independent contractor for Dysimmune Diseases Foundation, Boston MA, USA. P. Novak received speaker’s honoraria from KabaFusion, Cerritos, CA, USA and Lundbeck, Copenhagen, Denmark. P. Novak is a member of the Scientific Advisory Board of Endonovo Therapeutics, Woodland Hills, CA, USA. P. Novak obtained royalties from Oxford Press, Oxford, UK.
V. Lioutas has served as consultant for Qmetis, New York, NY, USA since 2020, unrelated to this project.
Sponsors
Novo Nordisk, Inc. and Medtronic, Inc. reviewed the manuscript, but had no participation in data analyses, manuscript preparation and submission decision.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Declaration of interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
REFERENCES
- 1.Chaudhari N, Ravi R, Gogtay NJ, & Thatte UM (2020). Recruitment and retention of the participants in clinical trials: Challenges and solutions. Perspectives in clinical research, 11(2), 64–69. 10.4103/picr.PICR_206_19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Walters SJ, Bonacho Dos Anjos Henriques-Cadby I, Bortolami O, Flight L, Hind D, Jacques RM, Knox C, Nadin B, Rothwell J, Surtees M, & Julious SA (2017). Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ open, 7(3), e015276. 10.1136/bmjopen-2016-015276. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Ross S, Grant A, Counsell C, Gillespie W, Russell I, & Prescott R (1999). Barriers to participation in randomised controlled trials: a systematic review. Journal of clinical epidemiology, 52(12), 1143–1156. 10.1016/s0895-4356(99)00141-9. [DOI] [PubMed] [Google Scholar]
- 4.Forsat ND, Palmowski A, Palmowski Y, Boers M, & Buttgereit F (2020). Recruitment and Retention of Older People in Clinical Research: A Systematic Literature Review. Journal of the American Geriatrics Society, 68(12), 2955–2963. 10.1111/jgs.16875. [DOI] [PubMed] [Google Scholar]
- 5.Mohseni S, Tabatabaei-Malazy O, Peimani M, Ejtahed HS, Khodaeian M, Nazeri E, Nouhi Z, Khodamoradi K, Aboeerad M, & Larijani B (2021). Withdrawal reasons of randomized controlled trials on type 2 diabetes: a systematic review. Daru: journal of Faculty of Pharmacy, Tehran University of Medical Sciences, 29(1), 39–50. 10.1007/s40199-020-00380-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Brueton VC, Tierney J, Stenning S, Harding S, Meredith S, Nazareth I, & Rait G (2013). Strategies to improve retention in randomised trials. The Cochrane database of systematic reviews, (12), MR000032. 10.1002/14651858.MR000032.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bower P, Brueton V, Gamble C, Treweek S, Smith CT, Young B, & Williamson P (2014). Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials, 15, 399. 10.1186/1745-6215-15-399. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Sood A, Prasad K, Chhatwani L, Shinozaki E, Cha SS, Loehrer LL, & Wahner-Roedler DL (2009). Patients’ attitudes and preferences about participation and recruitment strategies in clinical trials. Mayo Clinic proceedings, 84(3), 243–247. 10.1016/S0025-6196(11)61141-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Robinson KA, Dinglas VD, Sukrithan V, Yalamanchilli R, Mendez-Tellez PA, Dennison-Himmelfarb C, & Needham DM (2015). Updated systematic review identifies substantial number of retention strategies: using more strategies retains more study participants. Journal of clinical epidemiology, 68(12), 1481–1487. 10.1016/j.jclinepi.2015.04.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Honas JJ, Early JL, Frederickson DD, & O’Brien MS (2003). Predictors of attrition in a large clinic-based weight-loss program. Obesity research, 11(7), 888–894. 10.1038/oby.2003.122. [DOI] [PubMed] [Google Scholar]
- 11.Clark MM, Niaura R, King TK, & Pera V (1996). Depression, smoking, activity level, and health status: pretreatment predictors of attrition in obesity treatment. Addictive behaviors, 21(4), 509–513. 10.1016/0306-4603(95)00081-x. [DOI] [PubMed] [Google Scholar]
- 12.Sawamoto R, Nozaki T, Furukawa T, Tanahashi T, Morita C, Hata T, Komaki G, & Sudo N (2016). Predictors of Dropout by Female Obese Patients Treated with a Group Cognitive Behavioral Therapy to Promote Weight Loss. Obesity facts, 9(1), 29–38. 10.1159/000442761. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Novak V, Mantzoros CS, Novak P, McGlinchey R, Dai W, Lioutas V, Buss S, Fortier CB, Khan F, Aponte Becerra L, & Ngo LH (2022). MemAID: Memory advancement with intranasal insulin vs. placebo in type 2 diabetes and control participants: a randomized clinical trial. Journal of neurology, 1–19. Advance online publication. 10.1007/s00415-022-11119-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Galindo-Mendez B, Trevino JA, McGlinchey R, Fortier C, Lioutas V, Novak P, Mantzoros CS, Ngo L, & Novak V (2020). Memory advancement by intranasal insulin in type 2 diabetes (MemAID) randomized controlled clinical trial: Design, methods and rationale. Contemporary clinical trials, 89, 105934. 10.1016/j.cct.2020.105934. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Aponte Becerra L, Galindo Mendez B, Khan F, Lioutas V, Novak P, Mantzoros CS, Ngo LH, & Novak V Safety of Intranasal Insulin in Type 2 Diabetes on Systemic Insulin: A Double-Blinded Placebo-Controlled Sub-Study of Memaid Trial. Archives of Diabetes & Obesity, 4(2), 403–415. 10.32474/ADO.2022.04.000182. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Novo Nordisk Medical, Inc Novolin R (insulin human injection— Prescribing Information and Safety. Updated 11/2019; https://www.novo-pi.com/novolinr.pdf).
- 17.Cambridge Cognition Ltd. CANTAB Eclipse Test Administration Guide. Cambridge, England: Cambridge Cognition; 2012. [Google Scholar]
- 18.Wooten T, Sullivan DR, Logue MW, Fonda JR, Fortier CB, DeGutis J, McGlinchey R, Milberg W, & Esterman M (2021). Apolipoprotein E (APOE) ε4 Status Moderates the Relationship Between Close-Range Blast Exposure and Cognitive Functioning. Journal of the International Neuropsychological Society: JINS, 27(4), 315–328. 10.1017/S1355617720001034. [DOI] [PubMed] [Google Scholar]
- 19.Caballero J, Ownby RL, Jacobs RJ, Pandya N, Hardigan PC, & Ricabal LC (2018). Predicting medication adherence in older Hispanic patients with type 2 diabetes. American journal of health-system pharmacy: AJHP: official journal of the American Society of Health-System Pharmacists, 75(9), e194–e201. 10.2146/ajhp170067. [DOI] [PubMed] [Google Scholar]
- 20.Stubbs B, Schofield P, & Patchay S (2016). Mobility Limitations and Fall-Related Factors Contribute to the Reduced Health-Related Quality of Life in Older Adults With Chronic Musculoskeletal Pain. Pain practice: the official journal of World Institute of Pain, 16(1), 80–89. 10.1111/papr.12264. [DOI] [PubMed] [Google Scholar]
- 21.Abshire M, Dinglas VD, Cajita MI, Eakin MN, Needham DM, & Himmelfarb CD (2017). Participant retention practices in longitudinal clinical research studies with high retention rates. BMC medical research methodology, 17(1), 30. 10.1186/s12874-017-0310-z. [DOI] [PMC free article] [PubMed] [Google Scholar]