Bringing the real world to Psychometric Evaluation of Cervical Cancer Literacy Assessments with Black, Latina, and Arab Women in Real World Settings

Abstract

Objective

This research describes the development and evaluation of a new scale for assessing functional cervical cancer health literacy, the Cervical Cancer Literacy Assessment Tool (C-CLAT).

Methods

In Phase 1, 35 items in English, Spanish and Arabic, for C-CLAT were generated, taking into account three content domains-Awareness, Knowledge, and Prevention/Control. After content validation, 24 items were retained for psychometric evaluation. In Phase 2, the 24-item C-CLAT was evaluated in three racial/ethnic populations of urban women (N =543). Psychometric methods included item analysis, multifactor Item Response Theory modeling, and concurrent correlations.

Results

The final C-CLAT consisted of 16 items, with an internal consistency reliability of .72. C-CLAT reliabilities in Black, Latina, and Arab women were .73, .76, and .60, respectively. The rank order correlations of item difficulties across racial/ethnic groups was high (r’s = .97 to .98). The C-CLAT was positively related to educational level, and Arab women scored significantly higher than the Black and Latina participants.

Conclusions

This study presents a psychometrically sound instrument that measures health literacy related to cervical cancer.

Practice Implications

The C-CLAT is a tool that can be orally administered by a lay person and used in a community-based health promotion intervention.

1. Introduction

Health disparities are complex phenomena that involve different chronic diseases, various racial and ethnic populations, and many conceptual and structural barriers in addition to age, gender, socio-economic status, sexual orientation, and geographic factors. Adding to these complex phenomena and contributing to health disparity is health literacy (Parker, Baker, Williams, & Nurss, 1995; Gazmararian JA, et al., 1999).

Ratzen and Parker define health literacy as follows: “The degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions” (Ratzan & Parker, 2000). Thus, health literacy is, clearly, not solely based on one’s ability to read and write. When attempting to address the contribution of low health literacy to health disparity, two issues must be considered. First is the fundamental issue of health literacy itself: Is a person able to understand the medical information, in the language in which the directives are given, enough to participate accordingly? The second issue is the measurement of health literacy: What assessment tools are available to measure a person’s health literacy? Are these tools specific enough to distinguish between health literacy in one disease area and health literacy in another?

Our study addresses the latter set of questions about measuring health literacy. “Without measurement, it is impossible to know when and how health literacy may be relevant, and it would be very easy to design interventions that fail to attend to the relevant factor” (Paasche-Orlow & Wolf, 2010). In the current study, we used a definition of health literacy that focuses on functional cancer literacy. “… [C]ervical cancer literacy… [is] a woman’s functional understanding of her personal and familial risk of the disease, including how to minimize her risk and the risk to her family through preventive early detection screenings and life style changes and how to access the health care system and engage providers to minimize her risk and the risk to her family.” (Williams, Mullan, & Fletcher, 2007)

Cervical cancer is the third most common type of cancer in women (American Cancer Society, 2011) and is preventable. Although the HPV vaccine has been promoted as a protection against cervical cancer, at the time this study was conducted, its application was for teenage girls and not adult women who were the focus of this study.

The Health Information National Health Trends Survey estimated that most American women know that screenings for breast and cervical cancers are important, but they do not know the recommended ages at which they should be screened (National Cancer Institute, 2006). This is understandable given the frequency in screening guidelines due to new evidence. Since most female survey respondents were not sure when to get cervical cancer screenings, one might conclude that Americans’ health literacy is low in the area of cervical cancer screening.

Two of the most widely used health literacy tests are Rapid Estimate of Adult Literacy in Medicine (REALM), a health word recognition test, and Test of Functional Health Literacy in Adults (TOFHLA), a Cloze-style test of reading comprehension of health care material. REALM is the most commonly used word recognition test in medical settings (Doak, Doak, & Root, 1996; Davis, Michielutte, Askov, Williams, & Weiss, 1998). The high face validity of TOFHLA is corroborated by high positive correlations between test scores and patient education levels as well as negative correlations between test scores and age. TOFHLA item-total correlations have been found to be positive and consistent (coefficients range from 0.50s to 0.60s) within Black and English- and Spanish-speaking Latino/a groups (Aguirre, Ebrahim, & Shea, 2005). Moreover, TOFHLA has high test-retest reliability (Cronbach’s alpha 0.97) (p <.001) (Davis et al., 1993). REALM is highly correlated with other general reading tests and TOFHLA (coefficient 0.84) and has high criterion validity (Parker et al., 1995). TOFHLA has been widely used for health literacy research in medical and community settings. It has high content and criterion validity, with a validity coefficient of 0.84, with the reading section of REALM. Internal validity using Cronbach’s alpha is 0.98. While these are widely used and respected instruments, they were designed to measure overall health literacy and not functional cancer literacy (Williams et al., 2007).

It is important to understand that no one measure will be perfect for all potential uses (Pleasant, Mckinney, & Rikard, 2011). Those instruments that measure health literacy but are not disease-specific might not be appropriate for health literacy cancer prevention interventions. Using a combination of instruments might overburden the research participant (Williams, Templin, Hines, 2012).

2. Methods

The initial C-CLAT was a 12-item instrument and was tested in feasibility studies and culturally and linguistically translated (Rivera-Vasquez, Mabiso, Hammad, & Williams, 2009). The purpose of the second version of the C-CLAT was to increase the content representation in each domain and establish baseline functional cancer literacy with a larger culturally diverse sample.

In the current study we continued our qualitative formative work with three racial/ethnic groups, which included increasing the number of items, administering it to families, conducting focus groups with the participants and the community health workers who would be actually administering the C-CLAT. This allowed us to better capture cultural meanings and understandings of the items and make necessary revisions.

In November 2008, the Michigan State University Institutional Review Board provided approval for human subjects data collection with our community-based partners, the Detroit Department of Health and Wellness Promotion and the Arab Community Center for Economic and Social Services.

2.1 Research design

The Kin Keeper^SM Cancer Prevention Intervention was the vehicle used to validate the C-CLAT. The intervention is described in detail elsewhere (Williams, 2007). In brief, in the intervention, community health workers (CHWs) were contracted to recruit from their public health client caseload. There were seven CHWs for each racial/ethnic group. The race/ethnicity of the CHWs was the same as that of her clients. The CHWs administered the education intervention, which included the C-CLAT as a pre- and post-intervention instrument along with a sociodemographic questionnaire. The questionnaires were orally administered in the home of the kin keeper to her and her relatives. The family unit was defined as a “kin keeper” plus 2–4 bloodline female family members.

This sampling plan can be described as a three-level nested design in which family members are nested within their family unit (i.e., kin keeper unit) and family units are nested within CHW. This three-level nested structure was replicated in each racial/ethnic group—Black, Latina, and Arab, i.e., the nesting was within strata.

2.2 Participants

Participants were adult women who met the criteria we established for ”kin keeper” or “family member.” The “kin keeper” criteria were as follows: (1) aged 21 years or older; (2) participant in a CHW’s non-cancer-related public health program (e.g., maternal and child health); (3) self identified in one of the three racial/ethnic groups; (4) able to recruit, in any combination, 2–4 adult bloodline female family members (mother, sister, daughter, grandmother, aunt) to participate in two home visits; (5) willing to assist the CHW in locating the family members for the 12-month follow-up visit. The “family member” criteria were to be a kin keeper’s bloodline-related mother, sister, daughter, grandmother or aunt who was aged 21 years or older.

2.3 Procedure

With our established community-university partnerships, 21 CHWs were contracted through their agencies, 7 from each racial/ethnic group, to recruit a maximum of 10 of their clients and deliver the Kin Keeper^SM intervention. A recruited client agreed to recruit members of her adult female family, signed an informed consent form, and became known as the kin keeper.

Phase 1

The primary purpose of this formative phase was to expand and revise the C-CLAT. In the first step, a standard psychometric procedure, the C-CLAT was expanded from 12 to 35 items to narrow the scope of each question and devise the best linguistically and culturally appropriate items. The second step was to allow the CHWs to administer the C-CLAT to a set of kin keepers without delivering an educational intervention. The CHWs were instructed to note problem items. The sole purposes were for us to see which items worked and which were troublesome and to prepare the focus group protocol that allowed for salient discussion. The third step was to conduct six focus group sessions at the site of one of our community partners, three for kin keepers (N=12) and three for CHWs (N=21) based on their race/ethnicity. All three kin keeper focus groups were on one day back-to-back. The CHWs’ focus group sessions were held on a different day. The sessions were moderated by three experienced qualitative Black, Latina, and Arab researchers not associated with the study and each session had an observer to record non-verbal clues. Like the Latina and Arab CHWs, the Latina and Arab researchers were bilingual. The final step was data analysis.

Phase 2

We administered the Kin Keeper^SM Cancer Prevention Intervention to validate the newly revised C-CLAT. This form of instrument validation was done for two reasons: First, to simulate the real world environment in which the C-CLAT would be administered. Second, to address the issue of cancer disparities by providing these underserved women with practical education and resources for their immediate use. The C-CLAT was used as a pre- and post-education intervention tool. Since the CHW read aloud the C-CLAT in the family’s chosen language, we were able to measure the functional cancer literacy of all research participants regardless of their ability to read. In some instances, the Latina and Arab CHWs had to read in English and Spanish or Arabic to accommodate bilingual families. The home visitations lasted 1.5 – 2 hours.

2.4 Data analysis

Phase 1

Two investigators independently used triangulation, that compared and correlated the results from three different sources to one another (Cohen, l. & Manion, L. 1980). The analysis included data from focus group transcripts the observation notes and the notes from the CHWs during the home visits. Analysis compared the focus group data from the transcripts and observation notes with the CLATs completed at the home visits. Additionally, we conducted other tests to analyze the inter-item correlations, inter-item covariances, item mean, and item variances and total. As a result we were able to narrow the C-CLAT from 35 to 24 items in three languages, English, Arabic, and Spanish.

Phase 2

The objectives of the Phase 2 analysis were to: (a) examine the psychometric properties of the 24-item C-CLAT in a sample of 543 Black, Latina, and Arab women, (b) further refine the scale by deletion of under performing items, (c) determine scale reliability for each ethnic group and overall, and (d) examine evidence for concurrent validity. Due to the Kin Keeper^SM model protocol, observations were not independent. This dependency was taken into account in each of the analyses involving inferential statistics. Chi square tests of association used the Rao-Scott adjusted chi square statistic; the Item Response Theory (IRT) analysis was performed with Mplus, which adjusts standard errors for complex samples; and validity correlations were tested using multilevel modeling. Because Mplus could only account for one level of clustering, prior to analysis we examined the amount of scale variance attributable to both sources of sample dependence— community health worker and family membership. The variance attributable to CHWs (9.8%, p = .12) was not significant so we tested the multifactor IRT model with family membership, which accounted for 23.5% (p < .001) of the variance in the total score, as the cluster variable. Race/ethnicity was used as a stratification variable in determining standard errors.

The psychometric analysis was otherwise fairly routine. Item total and descriptive statistics were examined to exclude items with little or no variance from subsequent analyses. Confirmatory factor analysis was performed to determine if the dimensions used to generate the items corresponded to underlying psychological dimensions. Exploratory factor analysis was used to examine the latent structure basis of the item covariances in the event that the proposed dimensions might emerge with different item configurations. The best fitting model was refined, allowing correlation among some error terms and omitting additional items that did not achieve significant factor loadings. The internal consistency reliability of the total scale was computed for the overall sample and each racial/ethnic group separately. The rank order item difficulty across the three ethnic groups was examined. Finally, the correlation of C-CLAT with sociodemographic variables was examined.

3. Results

3.1 Sample

The sample consisted of 543 women in 176 families. Mean age was 45.70 (SD = 12.26). The kin keeper was slightly older (46.25 vs. 45.43) but this difference was not significant (p > .05). While 21 CHWs were contracted, 7 per racial/ethnic group, 2 Latina CHWs dropped out for personal reasons leaving 19 CHWs. Stratification by race/ethnicity resulted in 71, 50, and 55 kin keepers in the Black, Latina, and Arab groups, respectively. The percentage of participants in each group who were kin keepers was nearly equal: 33%, 32%, and 32% for Blacks, Latinas, and Arabs, respectively (chi square with 2df = .095, p = .95). There were exactly 3 members in 173 of 176 family units.

3.2 Item analysis

As a result of the item analysis, three items were omitted; one because of a large negative item-to-total correlation and two because 99% of participants answered the items correctly.

3.4 Multifactor IRT

These 21 items were subjected to a 3-factor confirmatory IRT factor analysis. The items were specified to load on the three factors that were used to generate the item pool. This model did not fit the data [χ² (70) = 255.29, p < .001; CFI = .76, RMSEA = .07]. Moreover, all factor loadings on the Awareness factor were less than .44 and not significant. (Note that the df for IRT models in Mplus using the WLSMV procedure is not the same as the number of free parameters, which was 45.) This confirmatory IRT analysis was followed up with an exploratory maximum likelihood factor analysis of the item tetrachoric correlations. Two-, three-, and four-factor solutions did not identify factors that resembled the three proposed. Three items which consistently showed small standardized factor loadings less than .2 were eliminated from subsequent analysis (24 − 3 − 2 = 19 items).

3.4 Final model

A one-factor 2-parameter IRT model allowing error covariance among items fit the 19 remaining items well [χ² (61) = 153.73, p < .001; CFI = .88, RMSEA = .05]. Three items with nonsignificant loadings were omitted, resulting in a final scale with 16 items and 9 residual covariances. The final model fit well [χ² (58) = 146.73, p < .001; CFI = .89, RMSEA = .05], and all loadings were significant (p < .05). The results are shown in Table 1, in the column “Std. Factor Loadings.” The four multiple choice items (6, 7, 15, and 16) had the strongest loadings, ranging from .77 to .86.

Table 1.

Cervical CLAT Item Statistics by Group and Final IRT Factor Analysis Results.

	Total Sample, N = 543			Racial/Ethnic Group
				Item Means
Item # and Item1	Mean	SD	Std. Factor Loading	Arab	Latina	Black
Awareness
1. Cervical Cancer is preventable [2]a	.73	.44	.29	.67	.65	.63
2. Cervical Cancer is a slow growing cancer [3}.	.65	.48	.27	.65	.64	.59
Screening & Knowledge
3. A woman should begin to have a Pap test after she becomes sexually active or when she turns 21, whichever comes first [5].	.81	.40	.43	.92	.85	.81
4. If a woman had a Pap test in the past with results that were abnormal, she does not need to continue getting regular Pap tests [7].	.79	.41	.29	.87	.84	.77
5. A woman is at risk for getting cervical cancer if she has unprotected sexual intercourse [9].	.56	.50	.19	.59	.57	.54
6. I should request a Pap test from my healthcare provider if I have [11]:	.71	.46	.77	.75	.72	.72
7. I should request a Pap test if [12]:	.73	.44	.79	.86	.83	.76
Prevention & Control
8. Using condoms decreases a woman’s chance for getting HPV infection that is the main risk factor for cervical cancer [13].	.66	.48	.28	.72	.66	.66
9. Precancerous changes and early cancers of the cervix generally do not cause pain [14].	.57	.50	.21	.54	.50	.50
10. When detected early, cervical cancer can be cured [15].	.89	.31	.49	.94	.87	.87
11. There are resources in my community for low and no cost cervical cancer screenings [16].	.82	.39	.45	.76	.73	.72
12. Getting a Pap test is very painful [20].	.69	.46	.53	.73	.66	.68
13. Women who do not have sexual intercourse do not need to get a Pap test [21].	.67	.47	.41	.67	.66	.64
14. Pap tests are for women in childbearing years, older women do not need it [22].	.77	.42	.46	.78	.74	.73
15. The following are risk factors for Cervical Cancer … [23].	.56	.50	.75	.62	.61	.57
16. Recovery from cervical cancer depends on [24].	.68	.47	.86	.74	.66	.68
Total % correct (Standard Deviation)	70.6	(19.48)		75.03	69.62	67.75
Internal consistency reliability	.72			.60	.76	.73

Notes.

These are abbreviated items. Complete list of items is in the appendix material.

^a[#] Numbers in brackets correspond to the numbers in the original item pool. The C-CLAT item pool had 24 items. Two items (#17, #18) had nearly 100% correct answers; three items (#10, #6, #19) had negative item-total correlations; and three (#1, #4, #8) had nonsignificant factor loadings. These 8 items were omitted leaving the final scale with 16 items.

3.5 Total scale score and reliability

The C-CLAT scale reliability (internal consistency, alpha) was .72, with a total percent correct of 70.6% (SD = 19.48) in the total sample. This score represents the percent of cervical cancer literacy knowledge attained at the time of testing. It is sufficiently low so that knowledge improvements can be monitored without ceiling effects.

3.6 Racial/ethnic group statistics

Table 1 also shows item means and scale reliability for each of the racial/ethnic groups. The rank order correlations of item difficulties across racial/ethnic groups was high (r’s = .97 to .98). The scale reliability was .73, .76, and .60 for Black, Latina, and Arab groups, respectively.

3.7 Correlations with sociodemographic variables

Mixed design analysis of variance was used to examine the relationship of sociodemograpic variables with the C-CLAT. Age group, marital status, income category, and type of health insurance coverage were not significantly related. The linear trend in educational attainment was significantly related to C-CLAT total [t(529.49) = −2.37, p = .02]. This trend is shown in Figure 1. Race/ethnicity was also related [F(2, 171.195) = 3.62, p = .029], with Arab women scoring significantly higher than either Black or Latina women (p < .05). Race/ethnicity remained significant after controlling for education (p < .02).

Figure 1.

Cervical CLAT total score by level of education with 95% confidence intervals.

4. Discussion and conclusion

4.1 Discussion

This study makes an important contribution to cancer disparities by providing a culturally appropriate health literacy assessment tool that measures functional cancer literacy. In our study, we revised and validated the C-CLAT, which measures functional health literacy specific to cervical cancer. The community based approach enabled this to be done in a real world setting. The psychometric analysis supports the use of the C-CLAT for use in Black, Latina, and Arab populations. Additionally, this study demonstrated that psychometric evaluation can be conducted within the context of community-based methods creatively reaching medically underserved women.

Engaging women, who are often removed from the health system, can be challenging. Working within a community structure was important. Vital was the use of CHWs who had the same cultural background and were able to administer the C-CLAT regardless of research participants’ reading ability in their first language. We used a pragmatic approach to evaluating the psychometric properties of the C-CLAT by measuring functional cancer literacy and then delivering the cervical cancer educational intervention, which gave participants immediate knowledge of resources available for cancer screening (Glasgow et al., 2012).

The instrument showed good internal consistency as well as reliability across each of the racial/ethnic groups of research participants. We note that the reliability for Arab women of .60 was acceptable (Wasserman & Bracken, 2003) but was lower than we would have liked. This fact does not diminish functionality of the instrument. It does open new opportunity to expand items that may improve the scale reliability. The rank order item difficulty was similar across the three groups.

4.2 Conclusion

The literature identified a need for more instruments that measure health literacy (Nielsen-Bohlman et al., 2004). By pschometrically evaluating the C-CLAT we have expanded the availability of health literacy instruments and, in particlular, those that measure functional cervical cancer literacy.

4.3 Practice implications

Validating the instrument with Black, Latina, and Arab women in English, Spanish, and Arabic provides baseline measures to assist in the development and refinement of cancer education and interventions in three distinct racial/ethnic groups of women.