Recruiting a representative sample in adherence research – the MALT multisite prospective cohort study experience

Abstract

Medication adherence is an important determinant of transplant outcomes. Attempts to investigate adherence are frequently undermined by selection bias: it is very hard to recruit and retain nonadherent patients in research efforts. This manuscript presents recruitment strategies and results from the MALT (Medication Adherence in children who had a Liver Transplant) multisite prospective cohort study. MALT sites recruited 400 pediatric liver transplant patients who agreed to be followed for 2 years. The primary purpose was to determine whether a marker of adherence, the medication level variability index (MLVI), predicts rejection outcomes. The present manuscript describes methods used in MALT to ensure that a representative sample was recruited, and presents detailed recruitment results. MALT sites were able to recruit a nationally representative sample, as determined by a comparison between the MALT cohort and a national sample of transplant recipients. Strategies that helped ensure that the sample was representative included monitoring of the outcome measure in comparison with a national sample, drastically limiting patient burden, and specific recruitment methods. We discuss the importance of a representative sample in adherence research and recommend that future efforts to study adherence pay special attention to sample characteristics.

Introduction

In clinical research, there are several factors that make it difficult to recruit a truly representative sample of the study population. Employing specific inclusion/exclusion criteria, by definition, introduces bias that differentiates the sample from an unselected clinic population¹. In addition to the inclusion/exclusion criteria, other attributes of a study mayalso lead to a biased sample; thoseare not necessarily described as selection criteria by the investigators². An example is the inherent bias in informed consent procedures. Patients who consent to research studies may be moremotivated by the hope that they will benefit from the research as compared with nonparticipants, and may be more altruistic than other patients; comprehension of consent procedures may factor into participation as well³. In pediatric research, this issue is even more complex as parents are the ones who are tasked with giving consent. Parents’agreement may reflect variables like comfort or obligation to the medical team⁴. Indeed, sampling bias has been well-documented in studies that require voluntary parental consent⁵. Another salient barrier to generalizability is study attrition. Attrition is rarely the result of a random process but rather is usually related to specific patient characteristics such as inability or lack of motivation to follow study procedures⁶.

The issue of selection bias is a concern in almost any area of clinical research^7,8, but effects of biases in recruitment are arguably most pronounced in research that is purporting to investigate adherence to medical recommendations as a way to improve medical outcomes (by improving adherence). Patients who are nonadherent to their medical care are also less likely to adhere to study procedures^{9, 10}. This leads to a situation in which much adherence research is conducted on the wrong group of patients – those who are relatively very adherent rather than nonadherent⁹. Because those adherent patients are less likely to suffer from adverse events, intervention studies often fail to show meaningful clinical gains even if adherence is somewhat improved¹⁰. In addition, conducting studies with mostly adherent patients might also lead to the development and advocacy of assessment or intervention techniques that are not likely to succeed with truly nonadherent patients. For example, elaborate adherence assessment methods (such as lengthy face-to-face interviews), or time-consuming interventions that involves frequent in-person meeting with subjects, are likely to be applicable only to the most adherent patients who are able to engage in those procedures⁹. Similarly, elaborate adherence monitoring devices, even when they do measure adherence accurately, are less likely to be used by the most nonadherent patients, and thus may provide little clinical benefit¹¹.

Selection bias in adherence research is therefore a major known confounder, not only in transplant medicine^12,13. But unlike many other areas of clinical care, pediatric transplant patients are followed very closely, almost exclusively in tertiary centers, and nonadherence is closely associated with poor outcomes¹⁴. The confluence of those factors suggests that an intensive focus on adherence in this population could be both very beneficial and feasible. In addition, transplantoutcomes data are routinely reported to national repositories such as UNOS (United Network of Organ Sharing, in the USA). This enablesa comparison between any recruited sample and a nationally representative repository. Transplant settings, therefore, provide a unique opportunity to focus on the recruitment of representative samples as well as examine the degree to which samples in research efforts are generalizable.

In this manuscript, we present the results of concerted efforts to reduce selection bias in the Medication Adherence in liver Transplant (MALT) study¹⁴, a prospective multisite cohort study of adherence in children who had a liver transplant. Within the limits of its inclusion/exclusion criteria, the MALT study aimed at recruiting a sample that is comparable, inasmuch as the prevalence of the study’s primary outcome (rejection episodes) is concerned, to a nationally representative sample of pediatric transplant recipients. We describe the methods used in MALT in order to try to achieve a representative sample and enhance retention, and present the results of those efforts. Following presentation of original results from the MALT cohort, we discuss specific strategies that can be applied – in the transplant field or more broadly - to ensure representative sampling in future research efforts.

Methods

The MALT study¹⁴ followed 400 pediatric liver transplant patients, each for two years, and examined the ability of a novel medication adherence measure, the Medication Level Variability Index (MLVI), to predict pre-defined medical outcomes. Rejection was defined in MALT based on a central pathology review of for-cause liver biopsies. The MLVI consists of a calculation of the standard deviation of medication blood levels in each patient. A higher standard deviation is observed in nonadherent patients and has been shown to predict poor transplant outcomes. This multisite (New York, Cincinnati, Pittsburgh, Los Angeles, Chicago) prospective cohort study was approved by respective Institutional Review Boards (IRB’s) of all sites and involved a full signed informed consent and an assent procedure when indicated (depending on the participant’s age). Inclusion/exclusion criteria have been described elsewhere¹⁴ and are summarized below:

Inclusion Criteria:

1. Age 1–17 years old at enrollment,

2. Receipt of a liver transplant at least one year prior to enrollment (we excluded recipients in the first year since transplant since variability in levels in this period is more likely to be affected by graft dysfunction and by variable prescription practices, not by patient adherence),

3. Tacrolimus is prescribed.

4. Participants had to be seen at the enrolling center at least once in the two years prior to enrollment, to ensure completeness of data.

Exclusion criteria:

1. More than one transplant (including bone marrow replacement),

2. Biopsy-proven rejection within the past six months from enrollment (to ensure that pre-existing rejection is not the immediate reason for fluctuation in medication levels),

3. Hepatitis C (as hepatitis C infection in transplant recipients might affect tacrolimus prescription practices),

4. Instructed by a physician not to obtain tacrolimus levels for at least one year,

5. Participants who were seen only for consultation (with most or all of the child’s routine care is provided at another center), to ensure that follow-up is occurring at the center of record,

6. Medically unstable/hospitalized at the time of enrollment (because of concerns about inability to provide informed consent/assent),

7. Participant or guardian who were actively psychotic or severely disoriented due to any cause, including hepatic encephalopathy (temporary exclusion) or had been diagnosed with moderate or severe mental retardation as defined in the Diagnostic and Statistical Manual of Mental Disorders, 4th edition (DSM-IV).

We engaged in the following efforts to ensure that the sample was representative and minimize attrition (by reducing patient burden):

Sampling

Study sites were chosen specifically to enable a robust representation of minorities: compared to national averages, Hispanic and black patients are relatively overrepresented in two sites (New York and Los Angeles) while underrepresented in some of the rest. The MALT sample’s racial/ethnic mix results are presented below.

Patient and parent burden

Patient and parent burden was limited to approximately 1 hour each at the initial clinic visit during which parents provided informed consent and both parents and patients completed psychosocial measures. The psychosocial measures packet was trimmed to include only very few questionnaires. Failure to fill in a questionnaire was treated as missing data – it was not required for participation, therefore failure to answer the questionnaires did not lead to patient withdrawal from the study. Patients were reimbursed for the time they spent during this initial visit. The study design explicitly did not include any additional burden on subjects or their parents, with all clinical follow-up data obtained as part of routine medical care (rejection outcomes, liver function tests) and collected from the medical chart.

Pre-defined checks for bias

Sites identified all potentially eligible patients and approached each eligible patient during their routine clinic appointment up to the target enrollment number. Sites kept a log of patients who were approached, not approached, consented, and did not consent. We compared the rate of rejection in the MALT cohort with the rate of rejection in a larger national cohort (North American data from the Studies of Pediatric Liver Transplantation cohort, SPLIT), adjusted for age, annually, and at the end of recruitment. This was to ensure that the rate of rejection (the MALT study primary outcome) in MALT was representative of that rate as reported in North America in general, over a similar time period. This additional scrutiny ensured that we were aware of any significant deviations in the rate of the Primary outcome before the study is over. If we were to observe a gap, especially an increasing gap, between MALT and SPLIT in this regard, we were ready to modify recruitment processes (for example, ask one site to recruit more or less patients, preferentially approach patients of different ages) in order to reduce that gap.

Final cohort characteristics analyses

With a HIPAA waiver, we obtained basic demographic information on each transplant center’s entire roster. We compared the rate of key demographic and transplant characteristics between patients who were potentially eligible but not approached for the study, patients who were approached but did not consent, and patients who consented. We also compared key characteristics in the final MALT cohort with the SPLIT matched cohort. Chi-Square tests were used to compare categorical variables and Kruskal-Wallis tests were used for numeric factors, with a predefined level of significance at p<0.05. For the primary comparison (rate of rejection in SPLIT vs. MALT), we used the following procedure. Children in the SPLIT database were first matched with MALT participants based on the following characteristics: recipients of only a liver transplant (not multiorgan recipients), at least 1 year posttransplant, age at transplant, gender and primary diagnosis. Rejections occurring during the post-transplant time period (either in an inpatient or outpatient setting), that coincided with a MALT participant’s follow-up in the study, were included in the comparison of rejection rates, expressed as number of rejections per subject-month.

Results

The recruitment diagram has previously been published¹⁴. Power analyses indicated a final sample size of 400, and recruitment proceeded until this number was reached. 644 patients were eligible at all sites, of whom 422 were approached consecutively; 401 agreed to be enrolled. Very few patients (5%) who were approached did not consent. One patient turned out to be ineligible upon further scrutiny (as he had a previously undocumented other transplant), for a final sample size of 400 patients. Attrition and premature study exit were extremely low and stood at 5% (expressed as number of patients, not patient-years) at the end of the study. Data from patients who were not able to participate in the entire study have not been used for the primary analysis.

A diverse ethnic/racial sample composition was achieved and is represented in Table 1, which is using the United States’ National Institutes of Health categories of race and ethnicity¹⁵. The MALT study reached its pre-defined sample size (400) 6 months ahead of schedule, as presented in the Figure.

TABLE 1.

MALT inclusion enrollment report, using NIH racial/ethnic definitions¹⁵

PART A. TOTAL ENROLLMENT REPORT: Number of Subjects Enrolled as of 03/27/2014 (Cumulative) by Ethnicity and Race
Ethnic Category	Female	Male	Total
Hispanic or Latino	54	42	96
Not Hispanic or Latino	153	144	297
Not reported	4	3	7
Ethnicity Category: Total of All Subjects	211	189	400
Racial Categories
American Indian/Alaska Native	0	1	1
Asian	15	10	25
Native Hawaiian or Other Pacific Islander	2	0	2
Black or African American	29	27	56
White	135	133	268
More Than One Race	3	0	3
Other	14	11	25
Not reported	13	7	20
Racial Categories: Total of All Subjects	211	189	400
PART B. HISPANIC ENROLLMENT REPORT: Number of Hispanics or Latinos Enrolled as of 03/27/2014 (Cumulative)
Racial Categories	Female	Male	Total
Black or African American	2	0	2
White	40	33	73
Other	1	2	3
Not reported	11	7	18
Racial Categories: Total of Hispanics or Latinos	54	42	96

FIGURE.

RATE OF RECRUITMENT IN MALT: NUMBER OF RECRUITED PATIENTS OVER TIME

A comparison of sample characteristics between patients who were approached and consented, not approached, and not consented, is presented in Table 2. There were no significant differences in age, race, gender, age at transplant, and reason for transplantation between the groups.

Table 2.

MALT sample characteristics, comparing between approached, not approached, and consented groups.

Characteristics	Approached, Not Consented	Not Approached	Consented	Total	P value
All Eligible Participants	21 (100.0%)	238 (100.0%)	400 (100.0%)	659 (100.0%)
Primary diagnosis
Missing	0	3 (1.3%)	0	3 (0.5%)	0.8667
Biliary Atresia	10 (47.6%)	110 (46.2%)	193 (48.3%)	313 (47.5%)
Other Cholestatic	2 (9.5%)	19 (8.0%)	35 (8.8%)	56 (8.5%)
Acute L. Failure	5 (23.8%)	24 (10.1%)	41 (10.3%)	70 (10.6%)
Metabolic	3 (14.3%)	36 (15.1%)	60 (15.0%)	99 (15.0%)
Tumor	0	19 (8.0%)	34 (8.5%)	53 (8.0%)
AIH	0	2 (0.8%)	3 (0.8%)	5 (0.8%)
Other	1 (4.8%)	25 (10.5%)	34 (8.5%)	60 (9.1%)
Age at transplant (years)
N	21	238	400	659	0.0147
Mean (SD)	3.2 (3.8)	2.5 (3.3)	2.9 (3.4)	2.7 (3.4)
Median (Min, Max)	2.5 (0.0, 13.0)	1.0 (0.0, 16.0)	1.3 (0.0, 15.8)	1.2 (0.0, 16.0)
Gender
Male	11 (52.4%)	116 (48.7%)	189 (47.3%)	316 (48.0%)	0.8593
Female	10 (47.6%)	122 (51.3%)	211 (52.8%)	343 (52.0%)
Race
American Indian or Alaska Native	0	0	1 (0.3%)	1 (0.2%)	0.1362
Asian	3 (14.3%)	6 (2.5%)	25 (6.3%)	34 (5.2%)
Black or African American	4 (19.0%)	24 (10.1%)	56 (14.0%)	84 (12.7%)
Native Hawaiian or Pacific Islander	0	0	2 (0.5%)	2 (0.3%)
White or Caucasian	9 (42.9%)	174 (73.1%)	268 (67.0%)	451 (68.4%)
Other	4 (19.0%)	19 (8.0%)	25 (6.3%)	48 (7.3%)
More than one race	0	2 (0.8%)	3 (0.8%)	5 (0.8%)
Not reported	1 (4.8%)	13 (5.5%)	20 (5.0%)	34 (5.2%)

There were no deaths in the MALT cohort. The rate of rejection per subject-month, adjusted for age and time since transplant, at SPLIT and MALT at the end of the two year period was identical, as previously reported, and stood at 0.005 for both cohorts¹⁴. A comparison of sample characteristics between the MALT and SPLIT reference population is presented in Table 3. There were no significant differences in age, race, gender, and reason for transplantation between the groups. There was a significant difference between age at transplantation in the SPLIT vs. MALT samples, with a slight over-representation of younger patients in MALT. 22.5% of MALT participants with evaluable levels had an MLVI > 2.5 at the end of the study period.

TABLE 3.

SPLIT vs. MALT: comparison of select characteristics between MALT and the reference North American SPLIT cohort.

Characteristics	Consented (MALT Participants)	SPLIT Participants	P Value
All Eligible Participants	400 (100.0%)	2469 (100.0%)
Primary diagnosis
Missing	0	6 (0.2%)	0.2530
Biliary Atresia	193 (48.3%)	1078 (43.7%)
Other Cholestatic	35 (8.8%)	239 (9.7%)
Acute L. Failure	41 (10.3%)	292 (11.8%)
Metabolic	60 (15.0%)	330 (13.4%)
Other	71 (17.8%)	524 (21.2%)
Age at transplant
< 1 year	169 (42.3%)	821 (33.3%)	<.0001
1–4 years	145 (36.3%)	840 (34.0%)
5–12 years	78 (19.5%)	574 (23.2%)
13–17 years	8 (2.0%)	234 (9.5%)
Gender
Male	189 (47.3%)	1158 (46.9%)	0.9140
Female	211 (52.8%)	1311 (53.1%)
Ethnicity
Missing	0	833 (33.7%)	0.2427
Hispanic or Latino	96 (24.0%)	384 (15.6%)
Not Hispanic or Latino	297 (74.3%)	1197 (48.5%)
Not reported	7 (1.8%)	55 (2.2%)
Race
Black or African American	56 (14.0%)	377 (15.3%)	0.2092
White or Caucasian	268 (67.0%)	1543 (62.5%)
Other	76 (19.0%)	549 (22.2%)

We evaluated recruitment figures, including site variation, biannually; the results were such that a stratification strategy was not necessary, even though we were ready to implement changes in recruitment if those seemed warranted at any time in the study.

Discussion

The MALT study mitigated selection bias by introducing design elements that we propose could be used in future adherence research. First, we selected sites based on likely patient characteristics. Second, we minimized patient burden by collecting all new (questionnaire) data in the first and only dedicated study encounter – and reimbursing patients for time spent in that encounter – while using only data from medical chart reviews in the 2 year follow-up period. Third, we identified and approached all potentially eligible patients sequentially as they came into clinic. Fourth, we monitored the characteristics of the recruited sample as compared with eligible patients who were not recruited. Fifth, we compared the rate of our primary outcome as well as basic patient characteristics with the rate of that outcome and those characteristics in a national sample: this procedure was feasible due to the existence of SPLIT database and might be uniquely suited to transplant medicine.

The issue of non-representative sampling in behavioral research is a known limitation.^7,16 Efforts have been proposed in the past as to how to address this bias, including, for example, reducing burden on participants and comparing participant to non-participant data¹³. The MALT adherence study is unusual in that it was able to recruit, and follow, a nationally representative, ethnically and racially diverse sample of patients in 5 centers. It reached its recruitment goals in less time than originally allocated, which may have meant that a higher ratio of eligible patients was approached in order to reach the recruitment goal quickly. We note that because of the consent requirement, we were unable to recruit patients who do not come to clinic at all. While this caveat may seem to suggest some level of bias against the recruitment of patients who are so nonadherent that they do not even come to clinic, in practice this almost never happens: complete failure to show to follow-up in transplant clinics in the participating centers could lead to a report to respective child protective agencies, who might assign a guardian to ensure that children receive care. Therefore, in practice it is exceedingly rare that children would miss appointments to the point that they are never seen. In MALT, attrition was very low, much lower than originally projected (5% rather than 25%, the original assumption, which was estimated based on the attrition rate for other cohort studies¹⁷). The characteristics of recruited patients vs. patients who were eligible but were not recruited were similar. The rate of the primary outcome in MALT was identical to SPLIT (our reference standard), and MALT sample characteristics were similar to SPLIT’s with one exception: more patients in MALT had a transplant at a young age (<1 year old) as compared with SPLIT. This could be because of a time effect, in that SPLIT sampling started before MALT, and transplanting younger patients has become commonplace only recently. Notwithstanding this small difference, it is safe to conclude that MALT was able to recruit and maintain a cohort that is fully representative of the national norms for pediatric liver transplant recipients. Since we only compared our results to North American data in SPLIT, a limitation of our approach is that our sample, while representative of North American patients, may not be representative of international cohorts; this caveat is important to bear in mind as the behavior of adherence, and its modifiers, are quite different between different nations¹⁸.

This achievement is not typical for adherence research, which almost always involves the recruitment of significantly non-representative samples¹⁹. For example, a recent adherence intervention trial in transplantation medicine reported excellent baseline adherence in the recruited sample (Medication Possession Ratio, MPR, of 0.83¹⁰.) The same authors, in a previous study, reported that the MPR rate in a much larger sample in their clinic as a whole was 0.53, a poor adherence rate²⁰. About a half of the patients who were approached did not participate in that intervention study, many because they did not return enrollment forms. Not returning enrollment forms reflects nonadherence to study procedures, so by definition that study selected against inclusion of nonadherent patients, which explains why the baseline rate of the primary outcome (adherence) in the study was much higher than the adherence reported for the non-selective sample. The statistically significant improvement in adherence in the intervention group (from a mean MPR of 0.83 to a mean MPR of 0.89 at the end of the study) probably did not lead to significant clinical advantages, given that the recruited sample was sufficiently adherent at baseline anyway. Indeed, some of the same authors later described a medication possession ratio of greater than 0.83 as representing adequate adherence²¹. In addition, because the intervention was primarily tested on adherent patients, it is still unclear whether the proposed intervention is feasible, or would be beneficial, in truly nonadherent patients. Improving adherence in patients who are nonadherent at baseline might require a completely different approach, as procedures that adherent patients are able to follow might not even be feasible in nonadherent patients.

The MALT study, in contrast, managed to recruit a substantial number of nonadherent patients (22.5% of the sample was determined to be nonadherent, a-priori defined as patients with an MLVI of 2.5 or above). The ultimate success of the MALT study should serve as proof that it is in fact possible to recruit representative samples in adherence research. However - as an observational cohort study, MALT investigators were able to design a minimally-burdensome protocol; it would be much harder to reduce burden as effectively in intervention research. We therefore believe that even if our methods are fully employed, they might not be sufficient to ensure a representative sample in intervention research. In those circumstances, we suggest that at the very least, the bias should be quantified and presented in as much detail as possible. For example – as we did –a comparison with another dataset (such as SPLIT) can be attemptedSuch a comparison may clarify the degree of bias in the intervention trial. Alternatively, another, entirely different approach may have to be used if a representative sample is simply impossible to achieve: pre-identification of nonadherence and targeting only nonadherent patients in the research. The MLVI marker’s pros and cons have been extensively discussed elsewhere¹⁴; MLVI, or any other method of assessment of adherence, can be used to design a study that will only recruit patients who are known to be nonadherent. The resulting “bias” (that the study will concentrate only on nonadherent patients) is beneficial in that the research will target only patients that need to be targeted. It is surprising that this kind of design is not used more frequently in adherence research, especially given that our preliminary data suggest that it is feasible²². MALT was not an intervention trial; our suggestions above are an effort to use MALT data to inform another type of research, and as such, those insights should be taken as tentative. Nevertheless, we believe that the strategy of recruiting only nonadherent patients into adherence intervention studies remains seriously underutilized in pediatric research in general (not only in transplant medicine). We would like to clarify that trying to recruit only nonadherent patients is not the same as recruiting patients who suffer from medical or psychosocial risks that may or may not be related to nonadherence. For example: focusing on patients with uncontrollable hypertension in behavioral intervention trials^23,24 does not qualify as recruiting “only nonadherent patients”. Increased or uncontrolled hypertension is a medical indicator of poor outcome, it does not measure the behavior of adherence. While recruitment of patients who are “in trouble” - suffer an increased medical or psychosocial risk – has certainly been tried before in multiple settings²⁵, it has shortcomings. Not all of those patients are nonadherent (the poor outcomes can be related to other issues). In addition, waiting until the medical poor outcome already happened²³ is not a desired selection criterion if one is trying to prevent medical complications rather than treat them after they have already happened. While choosing a “high risk pool” has been tried, we are not aware of any randomized controlled trials in adherence research in pediatric settings that had an inclusion criterion that specified that only nonadherent patients would be targeted for intervention. Indeed, a recent systematic review of adherence intervention studies, encompassing all pediatric chronic disease conditions, identified 42 randomized controlled trials, none of which specified nonandherence as an inclusion criterion²⁶.

In conclusion, the MALT observational cohort study was able to recruit a nationally representative sample, an unusual achievement in adherence research, using several design elements that are presented in this manuscript. If similar design elements are not used in adherence research, we propose that studies should be assumed to have substantial selection bias that selects for primarily adherent patients.