Alternative approaches to assessing intervention effectiveness in randomized trials: Application in a colorectal cancer screening study

Abstract

BACKGROUND

Previous analysis of a randomized community-based trial of a multi-component intervention to increase colorectal cancer (CRC) screening among Filipino Americans (n=548) found significantly higher screening rates in the two intervention groups compared to the control group, when using intent-to-treat analysis and self-reported screening as the outcome. This report describes more nuanced findings obtained from alternative approaches to assessing intervention effectiveness to inform future intervention implementation.

METHODS

The effect of the intervention on CRC screening receipt during follow-up was estimated using methods that adjusted for biases due to missing data and self-report, and for different combinations of intervention components. Adjustment for self-report used data from a validation substudy. Effectiveness within demographic subgroups was also examined.

RESULTS

Analyses accounting for self-report bias and missing data supported the effectiveness of the intervention. The intervention was also broadly effective across the demographic characteristics of the sample. Estimates of the intervention effect were highest among participants whose providers received a letter as part of the intervention.

CONCLUSIONS

The findings increase confidence that the intervention could be broadly effective at increasing CRC screening in this population. Subgroup analyses and attempts to deconstruct multi-component interventions can provide important information for future intervention development, implementation and dissemination.

INTRODUCTION

Asian Americans underutilize colorectal cancer (CRC) screening (1) but few interventions to increase screening have been developed and rigorously tested for Asian Americans. As in other immigrant populations, Filipino Americans who have a regular physician and health insurance, higher levels of education and income and who have been in the United States for a longer time period are more likely to obtain CRC screening than more recent immigrants who lack access to health care (2). We recently conducted a randomized colorectal cancer screening intervention trial among Filipino Americans. The trial used multiple intervention components with different participants receiving different combinations of components. We observed a significant increase in CRC screening by 6 month follow-up (25–30% in the two intervention arms versus 9% in the control arm), based on self-report and an intent-to-treat analysis that assumed a not screened outcome for participants lost to follow-up (3).

The purpose of this paper is to report in-depth analyses conducted to gain a more detailed understanding of the effect of the intervention. Articles reporting the primary outcome of intervention trials typically focus narrowly on reporting the results of the primary outcome analysis, using a single methodological approach, which often belies the complex realities of evaluating the effectiveness of a community-based intervention. For example, a recent review found that authors used a variety of strategies to deal with missing data in intent-to-treat analyses (4), and the robustness of findings to different strategies is not always clear. Other authors have noted the value of using multiple different analytic approaches, since discrepancies in findings could provide important additional information (5). The differential effectiveness of the intervention among segments of the population and identification of key components of an intervention are also important for a thorough evaluation of an intervention and its probable reach and impact.

Community-based intervention trials inevitably experience some degree of attrition of participants, and often must rely on self-reported outcomes. To obtain credible estimates, it is necessary to adjust for these common sources of potential biases. Accordingly, we obtained estimates of the effect of the intervention that used multiple imputation of missing values due to drop-outs rather than single imputation, and that took into account the results from a validation study of self-reported screening in a subsample of participants.

We also report subgroup analyses that examine whether the intervention was effective across the whole target population, or if it was differentially effective among subgroups with certain characteristics. This information can inform the future targeting of the intervention. Finally, we attempt to deconstruct the multi-component intervention to explore the efficacy of different combinations of intervention components. Estimating the efficacy of intervention components is important for future implementation and adaptation of the intervention for different populations. The identification of core components of an intervention -- components that significantly contribute to its efficacy -- is especially important if it is not feasible to implement all components in a particular setting or population.

METHODS

Details of the study design, randomization and intervention protocol have been described previously (3). Briefly, we recruited 548 Filipino Americans 50–70 years of age who were non-adherent to CRC screening guidelines (no FOBT within the past 12 months, no sigmoidoscopy within the past 5 years, and no colonoscopy within the past 10 years) at 45 Filipino American community-based organizations (CBOs) and churches. Subjects were randomized to one of three arms: small-group CRC education session with or without distribution of free FOBT kits or small-group education promoting physical activity (control arm). Participants were randomized in small groups of 6–10 subjects within the same CBO, to take advantage of existing bonds between members of the same organization. Telephone or face-to-face interviews were conducted at baseline and 6 months after the session in English or Filipino, and participants received a $20 incentive for each interview and a chance to win a $500 prize after completing the follow-up interview. The study was approved by the Institutional Review Board of the University of California, Los Angeles on 9/24/2004 and registered at ClinicalTrials.gov (NCT00742729).

Intervention

Guided by the Health Behavior Framework (6–8) we attempted to influence individual factors that are usually associated with cancer screening, including knowledge, attitudes, perceived barriers, and communication with provider about CRC screening (9, 10). The small groups of 6–10 participants met at community organizations for a 60–90 minute educational session to discuss incidence and mortality of CRC among Filipino Americans, benefits of prevention and early detection, recommended screening tests (FOBT, sigmoidoscopy, colonoscopy), where to obtain screening, and participant barriers to CRC screening. All participants were encouraged to discuss CRC screening with their physician. In one of the intervention arms, FOBT kits were passed out and participants without health insurance were instructed to mail FOBT kits to a community clinic that had agreed to process FOBT kits and charge them to the study. Participants in both intervention arms received print materials in English and Filipino (Tagalog) on CRC screening. Three months after the educational session, participants who had attended an intervention session were sent a personalized letter, reminding them to obtain an FOBT once a year. Physicians of these participants were informed by mail about their patients’ participation in our study and notified that they might submit to them a completed FOBT kit or ask for CRC screening. Of the 363 participants who attended an intervention session, 268 provided complete address information for their physician (74%). Due to an oversight, we failed to mail letters to physicians of participants who attended the first 25 intervention sessions. After we discovered the omission, we started mailing physician letters as planned for participants in the remaining 48 sessions. Thus, only 155 physicians received this letter out of 268 physicians for whom we had complete address information (58%).

Participants randomized to the control arm attended a small-group session in which they discussed physical activity. We conducted a total of 30 control group sessions. All educational sessions were conducted from 2005 to 2007 by one of 8 trained health educators who were recruited through the Philippine Nurses Association of Southern California.

Provider validation

Participants were asked for permission to contact their health care provider to confirm receipt of CRC screening during the study period. Of the 432 participants who completed the follow-up interview (79% of enrollees), 324 (75%) provided adequate contact information for their provider and signed a release form. Validation packets were mailed to providers of all 110 participants who self-reported any screening during the follow-up period and 98 randomly selected non-screeners. The provider validation package included a cover letter informing them of their patient’s study participation and a request for cooperation with the validation; an abstract of the study; a copy of the participant’s signed agreement allowing release of selected medical information; a form to record types and dates of CRC screening tests from the patient’s chart; a $20 gift certificate; and a pre-paid, return addressed envelope. We received provider validations for 142 participants (87/110, 79% of self-reported screeners and 55/98, 56% of self-reported non-screeners).

Statistical Analysis

The primary outcome was self-reported receipt of CRC screening (FOBT, sigmoidoscopy or colonoscopy) between the educational session and follow-up interview. In order to assess and adjust for potential biases due to missing data and self-report, we conducted outcome analyses using five approaches: (1) analysis of study completers that used self-reported outcomes; (2) analysis of study completers with self-reported outcomes adjusted for the positive predictive value (PPV) and negative predictive value (NPV) of self-report; (3) analysis of all randomized participants that used self-reported outcomes and handled participants without outcome data by singly imputing a not screened outcome (previously reported (3)); (4) analysis of all randomized participants that used self-reported outcomes and handled participants without outcome data using multiple imputation (11); and (5) analysis as in (4) with adjustment for PPV and NPV. The multiple imputation model included baseline variables deemed to be potentially prognostic for screening during follow-up, which were demographic characteristics, acculturation level, health care access, CRC screening history and previous doctor recommendation for screening, and study condition. This analysis was conducted using the MICE system of chained equations, implemented by the ice command in Stata. We obtained estimates adjusted for the PPV and NPV of self-report by randomly and multiply imputing provider-validated screening status based on self-reported status and PPV and NPV, where PPV and NPV were estimated by study arm based on the provider validation data. Analyses were conducted using mixed effects logistic regression with random intercepts to account for clustering by organization and session and were adjusted for language of baseline interview, which differed across study arms (p<.04).

We conducted subgroup analyses to estimate the efficacy of the intervention in subgroups defined by baseline demographic characteristics, acculturation level, health care access and CRC screening history by fitting models with intervention-by-subgroup interaction terms, with adjustment for language of baseline interview. The two intervention arms were combined, and Method 5 was used for these analyses. The subgroup analyses were not a part of the prospective analysis plan and thus are exploratory. Since the absence of a prospective analysis plan and alpha allocation makes p-values difficult to interpret (12), we do not claim statistical significance for these results but rather present point estimates and confidence intervals to gain a sense of effect sizes and their precision.

We explored the efficacy of combinations of intervention components by dividing the intervention participants into four groups based on receipt of a provider letter (yes/no) and an FOBT kit (yes/no) and comparing each group to the control group. These analyses were restricted to participants who provided sufficient information to enable a letter to be mailed to their provider and thus had the potential to receive this component of the intervention. Comparisons among the groups on baseline characteristics revealed differences (p<.05) in doctor recommendation for CRC screening, past history of CRC screening, health insurance status and whether the participant considered his/her self to be more Filipino or more American. To adjust for these differences, we computed propensity scores (13, 14) for the comparison of each intervention component combination group to the control group and estimated the intervention effect within quintiles of the propensity scores. Method 5 was also used for these analyses.

RESULTS

Participant Characteristics

All study participants were Filipino American immigrants. At baseline, the participants were, on average, 59 years old, had lived in the United States for 18 years, and 66% were female. Although most had health insurance (70%), only 25% had ever received any CRC screening test. Twenty percent had had an FOBT and 8% had had a sigmoidoscopy or colonoscopy, and none were current on screening per guidelines (which was an eligibility criteria). Participants in the control arm were more likely to have completed the baseline interview in English (69%) compared to participants in the two intervention arms (57%, p<.04). No other significant differences among study arms were found. More details are reported elsewhere (3).

Follow-up interviews were completed with 79% (432/548) of participants. Completers were more likely than non-completers to be college educated (72% versus 56%, p<.001), have an annual income of $50,000 or more (36% versus 24%, p<.02), and have completed the baseline interview in English (63% versus 51%, p<.01). Completers were also more likely than non-completers to report receipt of a doctor’s recommendation to obtain FOBT or an endoscopy at baseline (29% versus 19%, p<.03 and 32% versus 22%, p<.03).

Verification of self-reported screening

Table 1 displays the agreement between self-report and provider report for any CRC screening during the follow-up period. Overall concordance ranged from 73% in the intervention with FOBT kit arm and the control arm to 80% in the intervention without kit arm. The PPVs (proportion of screened self-reports confirmed by physician) were fair in the two intervention arms (62% and 72%), but very poor in the control arm (20%). The NPVs (proportion of “not screened” self-reports confirmed by physician) were excellent in all three arms of the study, ranging from 96% to 100%, with a single discrepancy in the control arm of an FOBT reported by a physician but not by the participant.

Table 1.

Comparison of participant self-report and provider report of any colorectal cancer screening (FOBT or sigmoidoscopy or colonoscopy) during the follow-up period (Filipino Health Study, Los Angeles, CA, 2004–2009)

Arm	Number of self reports of screening (A)	Number confirmed by provider (B)	Positive predictive value (B/A) (95% CI)	Number of self reports of no screening (C)	Number confirmed by provider (D)	Negative predictive value (D/C) (95% CI)	Concordance (B+D)/(A+C) (95% CI)
Intervention with free	45	28	62% (47%,76%)	18	18	100% (81%, 100%)*	73% (60%, 83%)
FOBT kit Intervention without	32	23	72% (53%, 86%)	14	14	100% (77%, 100%)*	80% (66%, 91%)
FOBT kit Control	10	2	20% (3%, 56%)	23	22	96% (78%, 100%)	73% (54%, 87%)

CIs obtained as binomial exact confidence intervals

^*One-sided, 97.5% CI

Efficacy of the intervention controlling for self-report bias and missing data

In a previous paper, we reported the efficacy of the intervention based on self-report and by singly imputing not screened for missing outcomes (Table 2, row 3, (3). Here we report alternative analytical approaches. The screening rates among study completers were 38%, 32% and 11% for participants assigned to intervention with FOBT kit, intervention without kit, and control condition, respectively, based on self-report (Table 2, row 1). After adjusting for PPV and NPV of self-report, estimated screening rates were markedly lower (row 2). However, the contrasts between the intervention and control groups as estimated by the odds ratios were as large (intervention with FOBT kit) or larger (intervention without kit).

Table 2.

Estimates of the efficacy of the intervention by five alternative analytic approaches (Filipino Health Study, Los Angeles, CA, 2004–2009)

		Approach			Percent screened			Intervention effect estimates
	Participants	Outcome variable		Adjusted for PPV and NPV of self-report?	Intervention with free FOBT kit	Intervention without FOBT kit	Control	Intervention with FOBT kit versus control		Intervention without FOBT kit versus control
		Study completers	Non-completers					OR (95% CI)	P	OR (95% CI)	P
1	Study completers (n=432)	Self- reported screening	N/A	No	38% (60/157)	32% (46/145)	11% (14/130)	5.4 (2.7, 11.0)	<.001	4.0 (2.0, 8.2)	<.001
2	Study completers (n=432)	Self- reported screening	N/A	Yes	24%	23%	6%	5.7 (1.7, 19.3)	.006	5.3 (1.6, 17.6)	.007
3*	All randomized participants (n=548)	Self- reported screening	Single imputation of not screened	No	30% (60/200)	25% (46/185)	9% (14/163)	4.8 (2.4, 9.8)	<.001	3.8 (1.8, 7.7)	<.001
4	All randomized participants (n=548)	Self- reported screening	Multiple imputation of self-reported screening	No	38%	31%	11%	5.4 (2.7, 10.7)	<.001	4.0 (1.9, 8.4)	<.001
5	All randomized participants (n=548)	Self- reported screening	Multiple imputation of self-reported screening	Yes	24%	23%	6%	5.7 (1.9, 17.1)	.002	5.5 (1.7, 17.9)	.005

OR, odds ratio; CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value. Results obtained using logistic regression with random intercepts to account for clustering on organization and session within organization and adjustment for language of baseline interview.

^*Approach #3 has been reported in Maxwell et al., 2010

Omission of study non-completers can break the prognostic balance between study arms created by randomization. This balance is preserved by the intent-to-treat analyses of all randomized participants reported in rows 3–5, which differ by method used to impute missing outcomes and adjustment for PPV and NPV of self-report. Screening rates estimated using multiple imputation for missing outcomes without PPV/NPV adjustment (row 4) were comparable to self-reported screening among completers (row 1), while rates estimated by singly imputing not screened for missing outcomes were lower (row 3). Adjusting for PPV and NPV of self-report decreased the estimated percentages screened in each group but heightened the contrasts between the intervention and control groups as estimated by the odds ratios (row 5).

All five analytic approaches yielded the conclusion that participants randomized to either intervention arm were significantly more likely to have been screened at follow-up than control arm assignees (all p<.025; Bonferroni adjustment for two comparisons per method). The approaches yielded estimated odds ratios for intervention with FOBT kit vs. control of 4.8 to 5.7, and for intervention without FOBT kit arm vs. control of 3.8 to 5.5. Estimated effects were larger after adjusting for the PPV and NPV of self-report, due in part to the low PPV of self-report in the control arm. The study was not powered to detect differences between the two intervention arms and we did not detect any differences.

Subgroup Analysis

Figure 1 provides results of subgroup analyses estimating the effect of the intervention (both intervention arms combined) in different segments of the participant sample. These analyses used multiple imputation of screening status for participants lost to follow-up and adjustment for PPV and NPV of self-report. The results suggest that the intervention was broadly effective across the demographic characteristics of the sample, with similar odds ratios in both genders, older and younger age groups, and participants with both higher and lower levels of education. The results are consistent with an intervention effect in both long-term residents who had lived in the U.S. for 20 years or more and those who had lived in the U.S. for less than 20 years. Of particular note, the estimated effect was similar for participants with and without health insurance, with or without a regular doctor, and for participants who had been screened in the past and those without a history of screening. The largest absolute difference in odds ratios was between participants who reported that a doctor had recommended CRC screening in the past (OR 7.7) and participants who had not received a doctor’s recommendation (OR 3.1). There was substantial overlap among all confidence intervals.

Figure 1.

Subgroup analyses, Filipino Health Study, Los Angeles, CA, 2004–2009

Analyses included all randomized participants and were conducted using mixed effects logistic regression with random for organization and session within organization, adjustment for baseline interview language, multiple imputation of screening status for participants lost to follow-up and adjustment for PPV and NPV of self-report.

Efficacy of Intervention Components

While most intervention participants received a small-group session, print materials and a reminder letter (N=363/385 randomized to either intervention arm, 94%), only about half received a free FOBT kit (by design) and letters were mailed to providers of only about 60% of participants who provided complete address information for their provider due to an oversight at the beginning of the study. Thus, most intervention participants received one of four combinations of intervention components: session/print/reminder, session/print/reminder/provider letter, session/print/reminder/FOBT kit, or session/print/reminder/provider letter/FOBT kit. We estimated the efficacy of each of these four combinations of intervention components compared to the control condition. Only participants who provided a valid address for their provider were included in this analysis to ensure that the comparisons were confined to participants who could have received this component of the intervention. In all, these analyses included 70% (113/163) of participants in the control arm and 70% (268/385) in the intervention arms.

As shown in Table 3, the screening rates in the two groups that had letters mailed to providers in addition to the small group education/print materials/reminder letter were significantly higher than the screening rate in the control group (27%/28% versus 5%). Odds ratios for these two combinations of intervention components ranged from 7.0 to 9.2 (both p<.05 compared to the control group). The screening rates among participants who did not receive a letter to the provider were about 10 percentage points lower and were not significantly different from the screening rate in the control group. However, failure to detect differences may have been due to the small sample sizes of these groups (N=64 and N=49). Screening rates were similar in the two subgroups that received a letter to the provider (27% and 28%) and in the two subgroups that did not receive a letter to the provider (18% and 19%), regardless of receipt of free FOBT kit. Odds ratios in the four intervention combination subgroups were not significantly different from each other. It should be noted that participants who were included in this comparison were more likely to have health insurance (81% versus 70% in the total sample), probably because the analysis was limited to participants who provided contact information for their provider.

Table 3.

Comparison of the efficacy of combinations of intervention components (Filipino Health Study, Los Angeles, CA, 2004–2009)

	Intervention combination subgroups				Control
	N=81	N=74	N=64	N=49	N=113
Intervention components
Small-group session, print materials and reminder letter	√	√	√	√
FOBT kit	√		√
Letter to provider	√	√
Estimated percent screened	27%	28%	18%	19%	5%
OR vs. control group	7.0 (1.5, 33.1)	9.2 (2.0, 42.7)	4.1 (0.7, 23.3)	5.3 (0.9, 32.0)
P-value	.015	.005	.106	.070

Analyses included all participants who attended a small-group session and provided sufficient information to enable a letter to be mailed to their provider. Analyses were conducted using mixed effects logistic regression with random intercepts for organization and session within organization, multiple imputation of screening status for participants lost to follow-up, adjustment for PPV and NPV of self-report and propensity score adjustment.

DISCUSSION

Using data from a randomized trial that tested a multi-component intervention to increase CRC screening among Filipino American immigrants in community settings, we conducted analyses using validation of self-report and alternative analytical approaches, subgroup analyses and analyses that attempted to deconstruct the multi-component intervention.

Self-reported versus provider-reported CRC screening

Validation of self-reported cancer screening through providers is challenging in community participants that have many different providers or no regular providers (15). Therefore, many studies rely on self-report (16–19). Although some studies have compared self-report to administrative and medical records (20), this information is lacking for Asian American populations. Similar to data from White, Black and Hispanic populations (21), we found fair to good PPVs in the two intervention arms of the study. However, only 2 out of 10 self-reported screenings were confirmed in the control arm. Although the numbers are small, this suggests that the screening rates in the control arm may have exhibited more self-report bias. NPV, on the other hand, was high in all study arms. This suggests that screening behavior was accurately reported by participants who did not get screened. Although neither self-report nor medical records are perfect, having both allows for better estimation of biases that may affect the results of studies.

Efficacy of the intervention by different analytical approaches

The reliability of the results of a randomized trial and in particular its effect size estimates depend on the extent to which potential sources of bias have been avoided or accounted for. We attempted to adjust our effect size estimates for two potential sources of bias common in community trial settings, attrition and the use of self-reported outcomes. Multiple imputation of outcomes for participants lost to follow-up yielded screening rates that were only marginally lower than rates computed for study completers. This is likely attributable to the fact that differences between completers and non-completers in baseline characteristics used to impute screening were small. It is important to note that multiple imputation relies on an assumption of missing at random, which cannot be directly tested for in the observed data. Adjustment for PPV and NPV yielded substantially lower overall screening rate estimates, but accentuated differences between the intervention and control arms as measured by the odds ratios. This may be attributable to the lower PPV of self-report among control participants compared to intervention participants. Comparisons with the control arm were statistically significant in all analytic approaches. While none of the assumptions incorporated in the different analytic approaches is likely to perfectly reflect the truth, the fact that we obtained similar results under a range of different assumptions provides confidence that the intervention effect was robust.

Our review of the cancer control literature indicates that there is no single standard analysis approach for reporting the outcome of randomized trials with respect to missing data. Some articles report intent-to-treat analyses that include only randomly assigned individuals with follow-up data (22, 23), and others report intent-to-treat analyses in which participants with missing follow-up data are conservatively coded as “not screened” (24). Others conduct a sensitivity analysis assuming a range of assumptions for outcomes for those lost to follow-up (25). Although we obtained comparable results using a variety of analytical approaches, other researchers may wish to consider conducting a similar set of analyses as we did in this report to evaluate the robustness of their findings.

Intervention effect in subgroups

While our subgroup analysis was post hoc, the overall pattern of results suggests that the intervention was broadly effective among the Filipino Americans in our sample, even among participants who had no health insurance and no regular provider, who would be hard to reach in a clinical setting. The fact that we had an arrangement with a community clinic that accepted FOBT kits from study participants and charged the study for the processing assisted those without insurance to get screened. The somewhat higher odds of screening among those who recalled a doctor’s recommendation underscores the importance of the doctor’s role in promoting any type of screening, which has been found in many other studies and populations (10, 19, 26, 27).

Efficacy of intervention components

Although we did not find statistically significant differences in the screening rates of participants who received different intervention combinations, point estimates were highest among participants who had a letter mailed to their provider, regardless of whether or not they received a free FOBT kit. This suggests that a letter to providers reminding them to recommend CRC screening to their patient is an important part of the intervention, possibly more important than distributing free FOBT kits. This analysis was restricted to participants who provided a valid address for their health care provider: 81% of participants that were included in this analysis had health insurance and were therefore able to request screening from their provider. This finding is important for future attempts to implement a similar program more widely in the Filipino American community and in other populations with high levels of health insurance.

Limitations

Participants in this study may not be representative of all Filipino American immigrants. Comparisons of self-reported and provider reported screening are based on small numbers, especially in the control arm, and participants who consented to provider verification may not be representative of all study participants. This may have biased findings that are corrected for PPV and NPV. Although medical records are often considered to be the “gold standard”, it is possible that screening tests were inaccurately recorded in patient records and/or inaccurately reported to us by physicians. In addition, small sample sizes for the validation study prohibited a separate examination of self-report and provider report for different CRC screening tests (FOBT versus endoscopy). The study was not specifically designed to investigate the effect of sending a letter to participants’ providers. Although we adjusted for known differences in the subgroups that received various combinations of intervention components, these groups may have been unbalanced with respect to unmeasured characteristics due to lack of random assignment and some groups were small.

CONCLUSIONS

This report provides an example, to encourage similar analyses for other cancer screening intervention trials. Studies should verify self-reported screening to enable adjustment for self-report bias. Subgroup analyses and attempts to deconstruct multi-component interventions may provide important information for future intervention development and implementation.