Factorial validity and invariance of four psychosocial constructs of colorectal cancer screening: does screening experience matter?

Abstract

Background

Few studies have examined the psychometric properties and invariance of scales measuring constructs relevant to colorectal cancer screening (CRCS). We sought to: 1) evaluate the factorial validity of four core constructs associated with CRCS (benefits, barriers, self-efficacy, and optimism); and 2) examine measurement invariance by screening status (currently screened, overdue, never screened).

Methods

We used baseline survey data from a longitudinal behavioral intervention trial to increase CRCS among U.S. veterans. Respondents were classified as currently screened (n=3,498), overdue (n=418), and never screened (n=1,277). The measurement model was developed using a random half of the sample and then validated with the second half of the sample and the full baseline sample (n=5,193). Single- and multi-group confirmatory factor analysis was used to examine measurement invariance by screening status.

Results

The four-factor measurement model demonstrated good fit. Factor loadings, item intercepts, and residual item variance and covariance were invariant when comparing participants never screened and overdue for CRCS, indicating strict measurement invariance. All factor loadings were invariant among the currently screened and overdue groups. Only the benefits scale was invariant across current screeners and never screeners. Noninvariant items were primarily from the barriers scale.

Conclusion

Our findings provide additional support for the construct validity of scales of CRCS benefits, barriers, self-efficacy, and optimism. A greater understanding of the differences between current and never screeners may improve measurement invariance.

Impact

Measures of benefits, barriers, self-efficacy, and optimism may be used to specify intervention targets and effectively assess change pre- and post-intervention across screening groups.

Introduction

Behavioral interventions that promote the uptake and maintenance of colorectal cancer screening (CRCS) must effectively address modifiable determinants of screening behavior. Over the past decade or so, a number of modifiable determinants have been identified (1, 2) and effective strategies to increase CRCS have been developed (3). Most of these modifiable determinants have come from health behavior theories and models; however, there is overlap in the conceptual definitions of constructs from different theories (e.g., confidence to perform a behavior is labeled self-efficacy in Social Cognitive Theory and perceived behavioral control in the Theory of Planned Behavior), and items used to measure the same construct are often not consistent across studies. The use of health behavior theories in interventions is recommended to advance our understanding of the theoretical mechanisms that drive behavior change (4). As a result, there is increasing interest in building consensus about theoretical constructs and health behavior theories and models (5–7). An important part of this effort entails establishing the validity of scales developed to measure psychosocial constructs. The use of valid and reliable measures is essential to examining the effects of behavioral interventions on psychosocial determinants of behavior change. Additionally, the use of standard measures allows for better comparison across studies.

Few studies have examined the psychometric properties of scales measuring constructs associated with CRCS (8–12). Vernon (8) examined five core constructs relevant to CRCS measured in a worksite intervention to increase CRCS among male automotive workers: salience and coherence, perceived susceptibility, cancer worries, response efficacy, and social influence. The five factor structure has since been replicated via confirmatory factor analysis with data from a U.S. urban primary care clinic (10) and a population-based sample in Ontario, Canada (11). McQueen (12) later provided evidence for the factorial validity of constructs measuring CRCS pros (i.e., perceived benefits), cons (i.e., perceived barriers), and self-efficacy in a clinic sample.

The psychometric characteristics of CRCS constructs must also be similar across various subgroups in order to make valid comparisons across different segments of the population. Of the studies that evaluated psychosocial constructs associated with CRCS, three have also examined measurement invariance (i.e., equivalence) across race and gender subgroups (10–12). Measurement invariance provides evidence of equivalence by examining the extent to which properties and interpretations of scale scores may generalize across population groups and settings (13). Collectively, studies of CRCS construct validity and invariance have contributed to the evidence base on establishing valid measures by demonstrating that important psychosocial measures of CRCS have similar psychometric properties in diverse settings.

To our knowledge, no study has tested measurement invariance that includes current screeners, overdue screeners, and never screeners in the same sample. We found only one study (12) that examined measurement invariance by screening status, and that study only included individuals who had never been screened or were overdue. Thus, it is unknown whether underlying latent constructs (e.g., barriers, self-efficacy) differ among those with different screening histories. Because past screening behavior is an important determinant of future screening behavior (1, 2), experience with CRCS, or lack thereof, may influence interpretation of and/or response to specific survey items in ways that make the measures of theoretical constructs less interpretable. A greater understanding of the similarities and differences, if any, in the experience and concerns of individuals with different screening histories may help us refine our measures and intervention targets.

We extend the findings of prior studies by evaluating the following specific aims: 1) examine the factorial validity of four constructs associated with CRCS (perceived benefits/pros, perceived barriers/cons, self-efficacy, and outlook/optimism); and 2) examine measurement invariance (i.e., equivalence) across subgroups of participants currently screened, overdue, or never screened. Measurement invariance among the screening subgroups will allow researchers to make valid comparisons of CRCS constructs, regardless of screening history. Because there is no consensus on the best approach to improve scales lacking invariance (14), we will focus on identifying the items with the strongest evidence of equivalence across groups and discuss several options for treating non-invariant items. Developing scales with strong psychometric properties is an iterative process of item refinement and evaluation. Establishing valid and reliable measures will advance our understanding and application of psychosocial constructs to interventions that effectively promote CRCS.

Materials and Methods

Setting

This research was conducted as part of a randomized controlled trial to promote CRCS among a population-based sample of U.S. veterans (clinicaltrials.gov identifier NCT01079533). The trial was approved by the institutional review board at the University of Texas Health Science Center at Houston. We used baseline survey data from the intervention trial.

Participants and Procedures

Our sample consisted of 5,287 U.S. veterans ages 50 to 64 that responded to a baseline survey and were potentially eligible for an efficacy trial. Study invitation letters and surveys were mailed between September 2008 and February 2010. We excluded participants that did not complete any of the psychosocial scales on the survey (n=94). The sample for this analysis consists of the remaining 5,193 respondents to the baseline survey.

Measures

The baseline survey was administered by mail and included questions on prior CRCS, as well as 35 items measuring four psychosocial constructs (perceived benefits/pros, perceived barriers/cons, self-efficacy, and optimism/outlook). Respondents were classified as being currently screened, overdue, or never screened based on U.S. Preventive Services Task Force and American Cancer Society guidelines (15, 16): 1) annual fecal occult blood test (FOBT); 2) sigmoidoscopy or double contrast barium enema within the last 5 years; or 3) colonoscopy in the last 10 years.

The scales evaluated in this study were adapted from our prior work (17). Perceived benefits, or positive aspects of the behavior, was measured with eight items, and perceived barriers, or negative aspects of the behavior, was measured using eleven items. Self-efficacy assessed confidence in performing certain aspects of CRCS with 10 items. Optimism was defined as a positive expectancy of the future. To our knowledge, optimism/outlook has not been extensively studied nor have its psychometric properties been evaluated in relation to CRCS. Dispositional optimism has been shown to be an important predictor of health maintenance behaviors (18). Researchers have increasingly become interested in examining how traits like optimism and defensive processing (19) may influence screening behavior. Six items were used from the Life Orientation Test-Revised as a measure of an individual’s expectations of good versus bad outcomes (20). Participants reported general optimism, not optimism related to CRCS or cancer. All items were measured on a five-point Likert scale ranging from “strongly disagree” (1) to “strongly agree” (5).

Data Analysis

Single- and multi-group confirmatory factor analysis was done using Mplus 7.0 (21) to test a four-factor a priori model. We used maximum likelihood estimation with robust standard errors (MLR) to account for the non-normality of the data (skewness: range −2.94—2.221, mean −0.37; kurtosis: range 1.59—10.97, mean 3.85). Full information maximum likelihood (FIML) was used to include respondents with missing data. FIML is preferable to other methods such as listwise deletion and single imputation (22, 23). There was less than 3% missing for any variable included in the analysis. Missing values for respondents with partial data ranged from 31 (“attpos”) to 112 (“bfind” and “btrans”). Several tests were used to evaluate the fit of the measurement models: χ²/degrees of freedom test, comparative fit index (CFI), and root mean square error of approximation (RMSEA) and its associated 90% confidence interval. Adequate to good fit is indicated when the CFI value is between 0.90—0.95 and above (24, 25). RMSEA values <0.07 suggest good model fit (25).

To examine factorial validity (Aim 1), we split the study population into two random samples. Using the first sample (n=2,598), we allowed for improvement and modifications with error covariances and by removing poor-performing items (i.e., low factor loading or cross-loading). The final correlated four-factor model was then confirmed with the second half of the sample (n=2,595), as well as the full sample (n=5,193).

To test measurement invariance (Aim 2), participants were divided into three screening subgroups: currently screened (n=3,498), overdue (n=418), and never screened (n=1,277). Using the measurement model developed in Aim 1, we performed independent, single-group confirmatory factor analysis on each of the three screening subgroups (baseline model). Multi-group confirmatory factor analysis was then used to determine if the model was invariant across screening subgroups: currently screened vs. overdue; overdue vs. never screened; and currently screened vs. never screened. We expected that invariance would be more likely between adjacent groups (i.e., currently screened vs. overdue) than extreme groups (i.e., currently screened vs. never screened).

Four models were created to test weak, strong, and strict measurement invariance as recommended by Wu (13) and Dimitrov (26). The first model (i.e., configural model) was unconstrained and allowed all factor loadings, intercepts, residual item variances/covariances, and factor variances/covariances to vary between groups. The second model tested metric invariance (i.e., weak invariance) by setting all factor loadings to be equal; the third model assessed scalar invariance (i.e., strong invariance) and constrained both factor loadings and item intercepts to be equal across screening groups. Finally, the fourth model tested the invariance of item uniqueness (i.e., strict invariance) by constraining the residual item variances and covariances to be equal. If any constraints produced significantly worse model fit, we examined the invariance of each scale separately. Similarly, we examined partial item invariance for factor loadings by successively releasing the constraints on individual items within scales that could not be fully constrained to be equal across groups.

For all tests of invariance, we used Satorra-Bentler χ² difference testing (27, 28) to evaluate the equality of covariance structures. Because the χ² statistic is often sensitive to large sample sizes, the Satorra-Bentler statistic incorporates a scaling correction factor for non-normal sampling distributions. We started with the least constrained model and subsequently imposed equality constraints of specific parameters across groups, thus producing nested models that are tested against each other using the Satorra-Bentler test. Invariance is indicated when the χ² value is not statistically significant (p > 0.05). Cronbach’s α was computed to evaluate the internal consistency reliability of the four scales. Alphas ≥ 0.70 were considered good (29).

A covariance matrix for the currently screened (Supplementary Table 1), overdue (Supplementary Table 2), and never screened (Supplementary Table 3) groups are available online as electronic supplementary material.

Results

Measurement Model

In developing the measurement model, the modification indices suggested that two benefits items (“benefits of screening outweigh the difficulties” and “screening tests are safe and have few complications”) cross-loaded with self-efficacy and barriers (Table 1). After adding cross-loading paths to the model, both items were confirmed to be significant cross-loaders and were dropped from the final model. To further improve overall model fit, and because each made conceptual sense, we then added three correlated error variances (“always optimistic” with “expect the best,” “complete CRCS even if nervous” with “complete CRCS even if embarrassed,” and “do not have insurance or copay is too high” with “too expensive”). All changes to the measurement model improved overall fit. The final correlated four-factor model was comprised of 33 items (6 benefits, 11 barriers, 10 self-efficacy, 6 optimism) and had acceptable fit to the data (Table 1). A similar fit was observed in the second half of the sample and the full baseline sample (Table 1). Cronbach’s alpha was above 0.80 for all of the scales, indicating good internal consistency (Table 2). Survey items and standardized factor loadings for the final correlated four-factor model are shown in Table 2.

Table 1.

Measurement model development and validation

Model Development	χ2	DF	SCF	p	CFI	RMSEA (90% CI)	Notes
First random half of sample (n=2,598)
All 35 items (8 benefits + 11 barriers + 10 self-efficacy + 6 optimism)	5210.653	554	1.4267	<0.001	0.883	0.057 (0.055—0.058)	Dropped “goben” cross-loaded with self-efficacy
7 benefits + 11 barriers + 10 self-efficacy + 6 optimism	4906.018	521	1.4136	<0.001	0.885	0.057 (0.055—0.058)	Dropped “gosafe” cross-loaded with barriers
6 benefits + 11 barriers + 10 self-efficacy + 6 optimism	4617.664	489	1.4016	<0.001	0.889	0.057 (0.056—0.059)	Added 3 correlated errors
6 benefits + 11 barriers + 10 self-efficacy + 6 optimism + 3 correlated errors	2943.515	486	1.3916	<0.001	0.934	0.044 (0.043—0.046)	Final model
Model Validation	χ2	DF	SCF	p	CFI	RMSEA (90% CI)
Second random half of sample (n=2,595)	3034.448	486	1.3823	<0.001	0.931	0.045 (0.043—0.046)
Full baseline sample (n=5,193)	4885.284	486	1.400	<0.001	0.933	0.044 (0.043—0.045)

NOTE: DF = degrees of freedom; SCF = scaling correction factor; CFI = comparative fit index; RMSEA = root mean square error of approximation; 90% CI = 90% confidence interval

Table 2.

Survey items and standardized factor loadings for the final correlated four-factor model using full baseline sample (n=5,193)

Survey Item	α	β
Barriers: In the future, I would NOT WANT to get tested for colon cancer because…	0.862
the test might find something wrong		0.584
it is too embarrassing		0.737
it is inconvenient or too hard to schedule		0.652
the stool blood test might be disgusting		0.659
a special diet or emptying my colon is too much trouble		0.650
it might be painful		0.713
I do not have symptoms		0.530
it is too expensive		0.539a
there is no one to drive me home from the test		0.494
it would be embarrassing to talk to my doctor about screening		0.677
I do not have insurance or the copay is too high		0.422a
Benefits: In the future, I would WANT to get tested for colon cancer because…	0.878
finding cancer early gives me a better chance at a cure		0.860
receiving clear findings would give me peace of mind		0.825
screening can find cancer early		0.896
my family would be happy if I got screened		0.541
getting screened is part of taking care of myself		0.819
if polyps are found and removed, cancer can be prevented		0.610
Self-Efficacy: How confident are you that you can…	0.964
make a decision about whether to get screened for colon cancer		0.646
complete colon cancer screening		0.869
complete colon cancer screening, even if you are nervous about it		0.912b
complete colon cancer screening, even if you are embarrassed about it		0.904b
complete colon cancer screening, even if you don’t think you need it		0.890
find time to complete colon cancer screening		0.893
talk to your doctor about colon cancer screening		0.855
complete any necessary preparation for colon cancer screening		0.902
get support from family and friends to help you complete colon cancer screening		0.733
complete colon cancer screening even if you think your health is good		0.912
Outlook	0.822
In uncertain times, I usually expect the best		0.489c
If something can go wrong for me, it will		−0.698
I’m always optimistic about the future		0.480c
I hardly ever expect things to go my way		−0.829
I rarely count on good things happening to me		−0.739
Overall, I expect more good things to happen to me than bad		0.581

NOTE: All factors are correlated (p<0.001): Barriers-Benefits, −0.203; Barriers-Self-Efficacy, −0.360; Barriers-Optimism, −0.308; Benefits-Self-Efficacy, 0.377; Benefits-Optimism, 0.116; Self-Efficacy-Optimism, 0.190. Factor loadings with the same subscript indicate items with significant error covariances (p<0.001). Factor loadings in bolded text were not invariant across currently screened and never screened subgroups.

Measurement Invariance

The configural models for the three screening subgroups showed acceptable fit (Table 3, Single-Group Confirmatory Factor Analysis). Invariance was observed when constraining factor loadings, item intercepts, and residual item variance and covariance for participants that were overdue or never screened, indicating strict measurement invariance (Table 3, Never Screened vs. Overdue). When comparing the currently screened and overdue subgroups, measurement invariance was supported when constraining the factor loadings only (i.e., metric invariance) (Table 3, Currently Screened vs. Overdue). The measurement model was not invariant across the currently screened and never screened subgroups (Table 3, Currently Screened vs. Never Screened). We conducted additional invariance testing that adjusted for sociodemographic variables that were modestly different between current screeners and never screeners, but this did not result in any significantly different findings (data not shown).

Table 3.

Results of the single-group and multi-group confirmatory factor analysis for the three screening subgroups (n=5,193)

Single-Group Confirmatory Factor Analysis	n	χ2	DF	SCF	p	CFI	RMSEA (90% CI)
Currently Screened	3,498	3523.253	486	1.446	<0.001	0.934	0.042 (0.041—0.044)
Overdue	418	1024.216	486	1.183	<0.001	0.919	0.051 (0.047—0.056)
Never Screened	1,277	2014.276	486	1.2757	<0.001	0.921	0.050 (0.047—0.052)
Multi-Group Confirmatory Factor Analysis
1. Never Screened vs. Overdue	1,695
1a. Unconstrained		3075.811	972	1.2294	<0.001	0.920	0.051 (0.049—0.053)
1b. Equal factor loadings		3121.509	1001	1.2262	<0.001	0.920	0.050 (0.048—0.052)
1c. Equal factor loadings and item intercepts		3177.309	1034	1.2191	<0.001	0.919	0.049 (0.048—0.052)
1d. Equal factor loadings, item intercepts, and		3157.673	1070	1.2626	<0.001	0.921	0.048 (0.046—0.050)
residual item variance/covariance
2. Currently Screened vs. Overdue	3,916
2a. Unconstrained		4797.438	972	1.3145	<0.001	0.933	0.045 (0.044—0.046)
2b. Equal factor loadings		4844.481	1001	1.3104	<0.001	0.932	0.044 (0.043—0.046)
2c. Equal factor loadings and item intercepts		5139.878	1034	1.2999	<0.001	0.928	0.045 (0.043—0.046)
2d. Equal factor loadings, item intercepts, and		5776.934	1070	1.3497	<0.001	0.917	0.047 (0.046—0.049)
residual item variance/covariance
3. Currently Screened vs. Never Screened	4,775
3a. Unconstrained		5631.944	972	1.3609	<0.001	0.931	0.045 (0.044—0.046)
3b. Equal factor loadings		5740.696	1001	1.3574	<0.001	0.930	0.045 (0.043—0.046)
3c. Equal factor loadings and item intercepts		6418.747	1034	1.3464	<0.001	0.920	0.047 (0.046—0.048)
3d. Equal factor loadings, item intercepts, and		7915.015	1070	1.3975	<0.001	0.899	0.052 (0.051—0.053)
residual item variance/covariance
Model Comparisons	χ2 diffa		DF diff		SCF diff		p
Never Screened vs. Overdue
1a vs. 1b	41.282		29		1.119		0.065
1b vs. 1c	45.692		33		1.004		0.070
1c vs. 1d	45.151		36		2.512		0.141
Currently Screened vs. Overdue
2a vs. 2b	35.786		29		1.173		0.180
2b vs. 2c	339.433		33		0.981		<0.001
2c vs. 2db
Currently Screened vs. Never Screened
3a vs. 3b	103.144		29		1.240		<0.001
3b vs. 3cb
3c vs. 3db

NOTE: DF = degrees of freedom; SCF = scaling correction factor; CFI = comparative fit index; RMSEA = root mean square error of approximation; 90% CI = 90% confidence interval; diff = difference

^aSatorra-Bentler χ² difference testing uses the normal-theory χ² test statistic divided by a scaling correction factor as a better approximation of a χ² distribution for non-normal data

^bSuccessive comparisons were untested when prior comparisons were non-invariant

When we further examined the invariance of factor loadings of individual scales and items among current screeners and never screeners, only the factor loadings for the benefits scale were invariant (p=0.066, data not shown). Although all item factor loadings ran in the same direction and were significantly different from zero, six barrier, two self-efficacy, and three optimism items were not equivalent between current screeners and never screeners (Table 2). These items were generally stronger indicators (i.e., larger factor loadings) of their respective construct for participants that were currently screened.

Discussion

Our results extend support for the factorial validity of previously published scales of CRCS benefits, barriers, and self-efficacy by including a subgroup that was currently screened. We also demonstrated evidence of factorial validity by replicating the measurement model in two random samples of our study population, as well as the full sample.

In addition, our study provides evidence of construct validity for an optimism/outlook scale. Researchers have previously hypothesized that dispositional optimism may overlap conceptually with self-efficacy (18), but our findings suggest that optimism is a distinct construct. Although both constructs have strong overtones of expecting a desired outcome, self-efficacy is often considered a state that is situational in relation to a specific behavior, whereas optimism may be considered a trait that is more stable across situations. This was reflected in our measures of the two constructs: the optimism scale was general in nature and the self-efficacy was specific to CRCS behavior. Self-efficacy and optimism were also modestly correlated in this study, which further supports their independence. Two recent reports found that dispositional optimism may moderate the influence of colorectal cancer worry and comparative risk on screening behavior (30, 31). Future research should examine how trait optimism may affect CRCS decision-making and whether optimism moderates the influence of CRCS perceptions and attitudes on behavior.

There are several implications of our findings. First, our study demonstrates invariance of factor loadings, item intercepts, and residual item variance and covariance (i.e., strict measurement invariance) when comparing subgroups that were never screened and overdue, as well as invariance of factor loadings (i.e., metric invariance) for all scales between the currently screened and overdue subgroups. Our finding of strict measurement invariance provides the necessary support for comparing the psychosocial characteristics of individuals with no CRCS experience and those that are overdue for screening. Strict measurement invariance is largely neglected in applied measurement practice (13, 26), yet is critical to establishing measures that can be used in diverse populations. We have provided evidence that all measures of CRCS benefits, barriers, self-efficacy, and dispositional optimism work well for those currently non-adherent to screening recommendations and eligible for intervention. Thus, our scales may be especially useful for intervention studies, which generally restrict eligibility to include overdue and never screened individuals.

Second, when comparing the most extreme subgroups, currently screened vs. never screened, only the factor loadings for perceived benefits were invariant. There is no consensus regarding a best approach to improve scales lacking invariance (14, 32); however, there are some recommendations that may be useful. For example, qualitative research could be used to identify barriers that are more important or salient to persons who have recently undergone screening compared to those who have never been screened or are overdue. In fact, previous research examining barriers to screening cited by current screeners and never screeners suggests that there may be important differences in how these two groups perceive CRCS barriers. Current screeners more often cite barriers associated with the screening test itself (e.g., fear of pain, dislike of prep solution), while never screeners are more likely to describe scheduling barriers (e.g., difficulty making an appointment, wait time) (33). In a mixed methods study, Jones et al. (34) found that focus group participants that had been screened were more likely to cite pain, discomfort, and trouble with anesthesia as barriers to screening compared with never screeners. A similar study that asked patients in primary care clinics to rank order barriers to CRCS also reported that respondents who had never been screened ranked time as a more important barrier compared to those up-to-date with screening (35). This is consistent with our finding that, compared with current screeners, survey respondents that had never been screened had lower factor loadings on items that measured barriers related to costs and experience.

Another option for improving invariance is to use a smaller subset of items found to be invariant across groups. In our study, the benefits scale was invariant among current screeners and never screeners. These and other invariant items may be used to compare mean scores across groups currently and never screened. Although this may prove to be a versatile approach in improving scales lacking invariance, the content validity may be reduced when only a few items are used. Finally, researchers may also take into account considerations other than statistical tests. For example, the statistical differences observed in the size of the factor loadings across groups in our study may be due to the large sample size and may not be substantive.

In summary, our study contributes to the literature that examines construct and factorial validity of psychosocial scales related to CRCS. This is especially relevant in light of recent calls for developing greater consensus about theoretical constructs used in health promotion studies. Our measures of perceived benefits/pros, perceived barriers/cons, self-efficacy, and optimism/outlook are appropriate for comparing current screeners vs. overdue and overdue vs. never screeners, and may be used to specify intervention targets and effectively assess change pre- and post-intervention across these groups. These scales should be examined in other studies of diverse populations. Further research using focus groups or cognitive interviews may also refine scales that measure barriers among current screeners and never screeners.