Validation of Diabetes Health-Related Quality-of-Life Instruments Using Cognitive Interviewing With Older African Americans

Abstract

The reliability and validity of two diabetes-specific quality-of-life instruments were examined with aging African Americans with limited literacy. Cognitive interviews with 15 persons and content analysis were conducted. Participants’ mean age was 72 years, and 40% had less than a high school education. Most observed problems with the surveys related to comprehension and response categories. Respondents had difficulty distinguishing between certain functional limitations and/or level of satisfaction as being associated with chronic illness or aging. This study underscores the need for research with special populations on the appropriateness of instruments previously validated and shown reliable for more general populations. While cognitive interviewing is resource intensive, ignoring this investment may result in inappropriate interpretations about the effectiveness of interventions to improve care and outcomes, understand differences, and eliminate disparities.

Older African Americans have a very high burden of type 2 diabetes, and a wide variety of medical and public health interventions have been designed and implemented to address this challenging chronic condition. Significant advances in prevention and health care have led to an increased average life expectancy, accompanied by a substantial proportion of aging adults living with chronic health conditions such as heart disease, diabetes, and cancer. The overarching goals of Healthy People 2010 are concurrently to increase life expectancy and to improve quality of life (QoL) as well as to eliminate disparities that exist among segments of our population (U.S. Department of Health and Human Services [USDHHS], 2000). Health-related quality of life (HRQoL) has gained significant attention as an important outcome measure. Outcomes research provides an evidence base that ideally drives informed decision making for program planners, patients, and clinicians (Agency for Healthcare Research and Quality [AHRQ], 2000).

Measurement of outcomes has historically focused on traditional biomedical markers and clinical end points. In diabetes care, such measures include A1C, lipids, and blood pressure levels. These clinical measures are clearly critical to assessing the impact of interventions. However, they fail to assess or acknowledge adequately the patient’s perspective or context, which may differ from the clinical parameters or proxy assessments made by health professionals (Woodend, Nair, & Tang, 1997). As public health proponents work to understand and eliminate apparent disparities in health and health care (Institute of Medicine [IOM], 2003), valid measurement of various process, impact, and outcome indicators are critical. HRQoL is one such indicator, and its use as a valid measure to be included in intervention studies is well established (Brown et al., 2004; Steed, Cooke, & Newman, 2003).

Few instruments purporting to measure HRQoL have been specifically pretested and validated in African American populations. Given the need for valid and reliable instruments for use within specific populations for research and to evaluate intervention, we undertook a qualitative study to better understand the appropriateness of existing HRQoL instruments with elder African Americans with diabetes. Study results from cognitive interviews and content analyses of two diabetes-specific HRQoL instruments are presented. Using an analysis coding system proposed by Willis, Schechter, and Whitaker, (1999), the following potential sources of cognitive process problems were examined: (a) comprehension/communication, (b) recall/computational, (c) bias/sensitivity, and (d) response category. Further, the effects of within-language cultural variations that may influence reliability and content or construct validity were considered.

BACKGROUND AND CONCEPTUAL FRAMEWORK

Definition, Dimensions, and Issues

HRQoL is an individual’s perception of his or her illness; it is that personal sense of physical and mental health and the ability to react to factors in the physical and social environments (USDHHS, 2000). A variety of complex factors operating at many levels (public policy, health system, practice, and individual) contribute to optimal health outcomes. With chronic disease, various personal, social, and environmental determinants affect HRQoL as conceptualized in a revision of Wilson and Cleary’s (1995) causal model of HRQoL (Ferrans, Zerwic, Wilber, & Larson, 2005). Several socioecological and cultural contextual models consider the environment, including elements such as medical care access, patient perceptions, and culture, and their impact on long-term outcomes such as HRQoL (Ashing-Giwa, 2005; Green & Kreutner, 2005; Wagner, 1998). Understanding the sociocultural context of individuals and systems affected by diabetes can inform and drive the development of interventions, measures, policies, and practices that are culturally relevant and maximally effective. Furthermore, understanding relationships among these factors could foster interactions between an informed, activated, and involved patient and a prepared, proactive health care team, with productive interactions that achieve desired functional and clinical outcomes (Barr et al., 2003; Wagner, 1998).

Clearly, HRQoL is a multidimensional construct, and, as such, instruments covering a broad range of dimensions—role performance, functional disability, financial concerns, and psychological and social well-being (Garratt, Schmidt, & Fitzpatrick, 2000; Rubin & Peyrot, 1999)—are best suited to measure this patient-centered outcome. Depending on how instruments are developed, subscores on specific domains as well as total summary scores may be calculated. Three types of HRQoL instruments (generic, situation specific, and disease specific) can be considered for use in studies of people with diabetes. Generic measures cover general aspects of health. Situation-specific measures are best known for assessing knowledge, attitudes, and beliefs. The third type, disease-specific measures, is the focus of this study. Diabetes-specific instruments focus on the condition, making them more clinic ally relevant and more responsive to changes in health (Garratt, Schmidt, & Fitzpatrick, 2002). Numerous generic, situation-specific, and disease-specific HRQoL instruments exist and have been reviewed elsewhere (Garratt et al., 2002).

An ongoing challenge is that no single measure comprehensively evaluates the many aspects of diabetes-specific HRQoL (Polonsky, 2000). Questioning whether an instrument covers salient aspects (i.e., content validity) as well as acknowledging that more questions equate to greater respondent burden are important considerations. Thus, when considering the “best” instrument, researchers and clinicians should evaluate the intended purpose as well as the appropriate fit for the population of interest. Other considerations include respondents’ age, education status, and severity of illness, all of which can impact the reliability and validity of HRQoL instruments (Rasin, 2004). Moreover, mode of administration and issues of selection bias are concerns, especially in certain high-risk groups, which include older populations, minority groups, and clients with higher levels of comorbidity (Unruh et al., 2003). It is established that self-administered surveys routinely result in a higher respondent burden; this burden is compounded for persons with low literacy levels. In addition, by relying solely on self-administered questionnaires, those who may benefit the most from intervention-based studies that evaluate HRQoL are underrepresented (Unruh et al., 2003).

Furthermore, health literacy—or how well an individual is able to obtain, process, and understand basic health information and services to make appropriate health decisions (IOM, 2003; USDHHS, 2000)—is linked to self-management and potentially to healthrelated outcomes. Patients with limited literacy skills are at risk for impaired patient–provider communication, inadequate health care, and adverse health outcomes, including worse health status (AHRQ, 2004). Notably, health literacy continues to gain recognition as an important variable associated with diabetes outcomes (Schillinger et al., 2002). Thus, multiple factors or determinants, such as literacy, cultural background, content of questionnaires, and mode of administration, need to be considered when evaluating the appropriateness of HRQoL measures, particularly in high-risk groups such as older African Americans.

Special Issues in Survey Pretesting

Instrument pretesting is an important element of survey design and evaluation and is best when it employs both quantitative and qualitative methods to identify potential sources of nonsampling error. Expert review, focus groups, interview debriefing, and cognitive interview are some of the established qualitative methods (Aday & Cornelius, 2006; Dillman, 2007). When considering possible contextual, intrapersonal, and personal differences that may impede eliciting reliable and valid survey responses, cognitive interviewing is an important research method and has been used as part of the evaluative process (Carbone, Campbell, & Honess-Morreale, 2002; Miller, 2003; Skelley et al., 2000).

Cognitive interviewing, grounded in cognitive psychology and information- processing theory, is used to generate, modify, or confirm contextually acceptable questions. Through the use of verbal probes, interviewers/researchers delve into cognitive and sociocultural processes associated with answering survey questions (Willis, 2005). As part of a multistage approach to questionnaire design, cognitive interviewing has become a broadly used method for pretesting and evaluating surveys to discover potential sources of error. As with between-language differences (e.g., English to Spanish), within-language differences can impact the cross-cultural reliability of a survey instrument. Within-language differences, those wordings or phrases that are unfamiliar within a specific region or community of people, may significantly contribute to comprehension and response category variation. Thus, semistructured interviews using verbal probing clarify that the phenomena under study are the phenomena intended for study; moreover, the probes delineate the ways in which the survey questions address these phenomena (Willis, 2005).

METHODS

Setting, Participants, and Recruitment

The study was undertaken from the fall of 2005 to the spring of 2006 in Charleston and Georgetown counties, South Carolina, after approval from the institutional review board of a health sciences center. Participants were recruited using flyers, church announcements, and word of mouth and from diabetes education classes held by REACH: Charleston and Georgetown Diabetes Coalition (Jenkins et al., 2004). Eligibility criteria included being African American, at least 55 years old, diagnosed with type 2 diabetes for more than 12 months, able to speak and understand English, community dwelling, and willing to complete two 1-hour, in-person interviews. This purposive convenience sample was used to ensure that those with lower formal education were included. We relied on lay community health advocates (CHAs) to introduce the study and refer potential participants. CHAs have a track record of trust and long-term relationships with community clients, thus acting as gatekeepers to potential participants. All potential participants were informed about the nature of the study, and authorization to audiotape each session was obtained. In exchange for their time, participants received a $40 gift card for each session completed. A purposive sample of nine female and six male African Americans was enrolled.

Instruments Tested

Two diabetes-specific HRQoL instruments were selected for study: the Audit of Diabetes Dependent Quality of Life (ADDQoL; Bradley et al., 1999) and the Diabetes Quality of Life Brief Clinical Inventory (DQOL-B; Burroughs, Desikan, Waterman, Gilin, & McGill, 2004). These particular instruments were selected on the basis of documented psychometric study, utilization, stated purpose, inclusion of items covering several QoL dimensions, and lack of reports on previous testing in southern African Americans.

The ADDQoL is a 19-item measure. Used with type 1 and 2 diabetes, the ADDQoL measures an individual’s perception of the impact of diabetes on QoL. Respondents are asked how a specific aspect of life would be better if they did not have diabetes, with response categories on a five-item scale (very much more, much more, a little more, the same, or less). Respondents are then instructed to rate the importance of each aspect of HRQoL, using a four-item response scale (very important, important, somewhat important, or not at all important). The two ratings are multiplied and summed to produce a final score. An in-depth report of the original ADDQoL has been reviewed elsewhere (Bradley et al., 1999). Although studies show the ADDQoL to be psychometrically sound (α = 0.84), the instrument has not undergone rigorous testing among African Americans, nor has it been tested as an interviewer-administered survey. The ADDQoL is traditionally a self-administered questionnaire; for this study, the authors, with permission from the developer, made selected wording changes to accommodate interviewer administration. The resulting modified version then underwent preliminary cognitive pretesting with two elders for face validity and flow prior to use in the study population.

The DQOL-B (Burroughs et al., 2004) is a 15-item inventory that yields a total score. The DQOL-B is a shortened version of the original 60-item Diabetes Quality of Life Measure (DQOL) treatment-focused instrument (Diabetes Control and Complications Trial Research Group [DCCT], 1988; Jacobson, de Groot, & Samson, 1994). Used with persons with either type 1 or 2 diabetes mellitus, the DQOL-B predicts self-reported diabetes care behaviors and satisfaction with diabetes control as effectively as the full-version DQOL. It was reported to be reliable (α = 0.85) and valid during the initial development and validation study of 498 people (Burroughs et al., 2004). However, the authors stated that testing in African Americans was needed, and longitudinal reliability and validity were unknown. The DQOL-B questionnaire, along with information about its development and validation, has been reviewed elsewhere (Burroughs et al., 2004).

Table 1 lists the QoL dimensions covered by items in each of the two HRQoL instruments and illustrates the variation in dimensions covered by each instrument. As noted earlier, diabetes impacts numerous aspects of life, and it would be impractical to attempt to cover successfully all dimensions on one instrument.

TABLE 1.

Dimensions of Health-Related Quality of Life by ADDQoL and DQOL-B Questions

Dimensions	Audit of Diabetes- Dependent Quality of Life	Diabetes Quality of Life Brief Clinical Inventory
I. Physical function Mobility, dexterity, range of motion, physical activity, activities of daily living, ability to eat whenever and wherever, physical ability, ability to operate machinery and drive a car, perform basic self-care	Q5—Physical ability
II. Symptoms Diabetes-specific (hypoglycemia, polyuria, thirst), general symptoms (pain, nausea, energy, sleep)		Q7—Sleep Q10—Treatment Pain Q12—Physically ill
III. Global judgments of health General QoL or impact of diabetes on general QoL, ability to pursue own schedules or plans, where expressed as a general issue	QI—General QoL QII—Diabetes general QoL Q20—Diabetes, self-management, and complication on QoL Q17—Depend on others
IV. Psychological well-being Anxiety, depression, coping, positive well-being and adjustment, sense of control, self-esteem, aggression, moodiness, worry— disease worry, social/vocational worry (family matters, the future, and job-related issues)	Q11—Self-confidence Q12—Motivation Q14—Future worries	Q4—Worry (vocational) Q13—Worry (disease)
V. Social well-being Family and intimate relations, social contacts, integration and social opportunities, leisure activities, sexual activity and satisfaction (effect on leisure activities and social relationships), ability to travel	Q1—Leisure Q4—Vacation Q3—Travel Q6—Family Q7 and Q8—Friendship Q9—Sex Q17—Depend on others Q18—Freedom to eat Q19—Freedom to drink	Q8—Sex Q11—Family (burden)
VI. Cognitive functioning (rarely included) Cognition, alertness, concentration, memory, confusion, ability to communicate	—	—
VII. Role activities Employment, household management, financial concerns, schooling	Q2—Work Q15—Financial Q16—Living situation/condition	Q9—Career
VIII. Personal constructs Satisfaction with body appearance, stigma and stigmatizing conditions, life satisfaction, spirituality Perceptions and behaviors of others, hide the existence of diabetes or treatment from others, reactions of others related to disease	Q10—Physical appearance Q13—Stigma (others)	Q3—Diet (others)
IX. Satisfaction with care and flexibility of treatment Clearly relates to satisfaction and not an aspect of QoL (e.g., if refers to one major construct such as social life, it would be classified under social well-being), diet, time needed for care, including doctors and consultations		Q1—Treatment Q2—Time: managing Q5—Time: monitoring Q6—Time: exercising Q14—Time: checkups
Other		Q15—Knowledge

Note. Question number (Q_) represents the corresponding item number of the instrument followed by a brief description of question topic. Source: Adapted from “Classification System of Dimensions for Patient-Assessed Health Outcome Measures,” Fitzpatrick, R., Davey, C., Buxton, M. J., & Jones, D. R. (1998). Evaluating patient-based outcome measures for use in clinical trials. Health Technology Assessment, 2(14), 1–74 (1998), and Garratt et al. (2000).

Measures of Participant Characteristics

Demographics

Demographic information collected during the initial interview included gender, birth year, education level, and income ranges. Other self-reported personal characteristics included comorbidities, assessment of general health status, mode of diabetes treatment, and marital and employment status.

Health Literacy

The Rapid Estimate of Adult Literacy in Medicine (REALM; Davis et al., 1993) and the Short Form Test of Functional Health Literacy in Adults (S-TOFHLA; Baker, Williams, Parker, Gazmarian, & Nurss, 1999) were used to assess health literacy. The REALM, historically the most commonly used instrument, is a word recognition test consisting of 66 medical terms that respondents read aloud to an interviewer. One point is given for each correctly pronounced word; scores vary from 0 to 66. The S-TOFHLA, another widely used measure, is a 7-minute timed reading comprehension test that uses the modified Cloze procedure (Taylor, 1953), where every fifth to seventh word in a passage is omitted. The respondent selects a word to fit into the blank from the four multiple-choice options provided for each space in the passage. Scores are scaled and range from 0 to 36. Scores assign health literacy skills into four grade-equivalent reading levels for the REALM and either adequate, marginal, or inadequate health literacy for the S-TOFHLA. An explanation of the scores is presented later in Table 7.

TABLE 7.

Literacy Scores and Selected Demographic Characteristics of Respondents With Most Problems on Both Instruments

Age/Gender	Number of Problem Questions		Grade Level Attained	REALMa	S-TOFHLAb	Years Since Diagnosis
	ADDQoL	DQOL-B
73/female	10	8	5th	7	6	30
82/female	4	7	3rd	36	0	25–30
64/male	6	4	11th	27	15	12
79/female	4	3	9th	26	5	31
66/male	3	3	10th	7	18	33
REALM Score Interpretationa
Raw Score	Grade Range Equivalent		Literacy Skills
0–18	3rd grade and below		Will not be able to read most low-literacy materials …
19–44	4th to 6th grade		Will need low-literacy materials with accompanying verbal instructions …
45–60	7th to 8th grade		Will struggle with many patient education materials and may be able to pronounce but not fully comprehend the message …
61–66	High school		Will be able to read most patient education materials …
S-TOFHLA Functional Health Literacy Levelsb
Score	Level		Functional Health Literacy Description
0–16	Inadequate		Unable to read and interpret health texts
17–22	Marginal		Has difficulty reading and interpreting health texts
23–36	Adequate		Can read and interpret most health text

^aREALM score interpretation (Davis et al., 1993).

^bS-TOFHLA Functional Health Literacy Levels (Baker et al., 1999).

Data Collection

Semistructured interview guides were constructed, and the interview protocol was formulated using a modified version of a Question Appraisal System (Willis, 2005) to construct a hybrid of standard and expansive verbal probes (Beatty, Schechter, & Whitaker, 1996) shown in Table 2. Examples of verbal probes from the interview guides are shown in Table 3. The interview guides also included a brief explanation of probing to familiarize respondents with the cognitive interview process. All interviews were digitally audiotaped. Additionally, the interviewer took abbreviated field notes. Following each session, the field notes and audio recordings were reviewed to identify potential problems with questionnaire, probes, and interview guides. This iterative process allowed interviewer and analysts an opportunity to address needed issues prior to or during subsequent interviews.

TABLE 2.

Using a Question Appraisal System to Develop Standard Cognitive Probes

Instructions: Conflicting or inaccurate instructions Complicated instructions Probe—Before I get to the actual question, tell me what this introduction is telling you.

Clarity: Problems related to communicating the intent or meaning of the question Wording: Question is lengthy, awkward, or ungrammatical or contains complicated syntax. Probe—Can you tell me in your own words what that question was asking? Technical terms: Undefined, unclear, or complex. Probe—What does the word [term] mean to you as it's used in this question? Vague: Multiple ways to interpret the question or to decide what is to be included or excluded. Probe—Tell me what you were thinking when I asked about [topic]. Reference periods: Missing, not well specified, or in conflict. Probe—Can you remember what time period the question was asking about? Probe—You said [answer], What time period does that cover?

Assumptions: Determine if there are problems with assumptions made or the underlying logic Inappropriate assumptions: Made about the respondent or about his or her living situation. Probe—How well does that question apply to you? Probe—Can you tell me more about that? Assumes constant behavior: Or experience for situations that vary Probe—Would you say that mostly stays the same, or does it vary or depend? Double-barreled: Contains more than one implicit question. Probe—Tell me more about your opinions on that.

Knowledge/memory: Check whether respondents are likely to not know or have trouble remembering information Knowledge may not exist: Respondent is unlikely to know the answer to a factual question. Probe—How much would you say you know about [topic]? Attitude may not exist: Respondent is unlikely to have formed the attitude being asked about. Probe—How much thought would you say you've given to this? Recall failure: Respondent may not remember the information asked for. Probe—How easy or difficult is it to remember [topic]? Probe—You said [answer]. How sure are you of that? Computation problem: The question requires a difficult mental calculation. Probe—How did you come up with that answer?

Sensitivity/bias: Assess questions for sensitive nature or wording for bias Sensitive content (general): The question asks about a topic that is embarrassing or very private or that involves illegal behavior. Probe—Is this OK to talk about in a survey, or is it uncomfortable? Probe—In general, how do you feel about this question? Sensitive wording (specific): Given that the general topic is sensitive, the wording should be improved to minimize sensitivity. Probe—The question uses the word [term]. Does that sound OK to you, or would you choose something different? Socially acceptable response is implied by the question. Probe—How did you come up with that answer? Probe—Do all the possible answers here seem OK, or did it seem like there's one that's supposed to be the right answer?

Response categories: Assess the adequacy of the range of responses to be recorded Open-ended question Probe—Was it easy or difficult to decide what answer to give? Mismatch Probe—How easy or hard was it to find your answer on that list? Probe—You said [answer], How well does that apply to you? Technical terms Probe—In this list, what does [term] mean to you? Vague Probe—Tell me what you were thinking when I asked about [topic]. Overlapping Probe—How easy or hard was it to choose an answer? Probe—Tell me why you chose [answer] instead of some other answer on the list. Missing Probe—How easy or hard was it to choose an answer? Illogical order Probe—How was it for you to go through that list? Did that cause any difficulties?

Other problems: Not previously identified Probe—Tell me more.

Source: Adapted from Willis (2005, table 5.1, pp. 81–83).

TABLE 3.

Example of Probes Used in Interview Guides

General neutral	Tell me more. How do you mean?
Question specific	To you, what does it mean to have a bad night’s sleep because of diabetes? How did you determine the amount of time spent getting checkups? Is this question important to you? Is this OK to talk about in a survey, or is it uncomfortable? When you hear the words “close personal relationship, ” who or what do you think about? Do you use another word or words to describe a “close personal relationship”?
Multiple questions	Tell me what you were thinking when I asked about […] How easy or hard was it to choose an answer? Tell me in your own words what this question is asking.

The interview location was based on participants’ preference, either in their homes or in a private room within a community center or clinic setting. One same-race interviewer trained in cognitive interview techniques conducted all the interviews. Instrument order was randomly assigned.

Data Analysis

Analytic approaches for cognitive interviews can vary greatly, but Willis (2005) offered seminal guidance in documenting and analyzing interview data. Building on the work of Tourangeau (1984), Willis (2005) and Willis et al. (1999) proposed four problem types in cognitive processing and subsequently developed a coding system for formal analysis of probed interviews. The first problem type, comprehension/communication, is an encoding process in which participants do not understand the terminology used. A recall/computational problem is a retrieval process wherein respondents have difficulty remembering a selected period of time (e.g., over the past 6 months). Issues related to bias/sensitivity are connected to respondents’ judgment process, such as “The question makes it sound like you have an extramarital relationship.” Finally, response category difficulties are created when respondents are unable to connect their own responses to the answer categories provided by the instrument. These four cognitive processing dimensions provided a framework for analyzing the data.

Digital audiotapes were analyzed following each interview. Field notes were used to clarify and highlight key points in the audio transcripts. To organize participant responses and facilitate analysis, modified interview guides were used as data entry and analysis forms. Transcripts of each of a participant’s two interview sessions underwent content analysis. Each subject’s interview sessions (two) were then reviewed for within-subject observations. Next, the findings were aggregated across all subjects, producing a summarized question-by-question analysis of the results associated with each instrument. This step allowed the analyst to look for overlapping problem areas that surfaced from each cognitive interview session matched to an individual subject. In other words, were there paired differences, and what, if any, areas overlapped? Finally, aggregated interview data were analyzed to elucidate problem trends or themes that occurred consistently across interviews for the ADDQoL and the DQOL-B. Characteristic quotes have been selected to exemplify identified issues. Although some quantification (frequency) of a problem was tracked, such data should be analyzed cautiously, and inferences should be limited. Problems identified during cognitive interviews should inform teams as to what may occur in the field environment versus what was observed in a small cognitive interview sample (Willis, 2005).

RESULTS

Characteristics of Participant Demographic

Table 4 reports participant characteristics. The mean age was 72 years (range 57–86 years), 40% had less than a high school diploma, and the majority reported a current income of less than $20,000. Most were retired, living alone, had multiple comorbidities (such as hypertension and heart disease), were taking diabetes medications, and rated their health as good.

TABLE 4.

Self-Reported Personal and Demographic Characteristics (n = 15)

Variable	n (%)
Gender
Male	6(40)
Female	9(60)
Age
Mean (years)	72
Range (years)	57–86
Education
Elementary (< ninth grade)	3(20)
Some high school	3(20)
High school diploma/general equivalency diploma	6(40)
Some college	2(13)
College (4+ years)	1(7)
Employment status
Retired	8(53)
Self-employed	1(7)
Unable to work	6(40)
Annual household income
<$10,000–$14,999	5(33)
$15,000–$19,999	6(40)
$20,000–$24,999	1(7)
$25,000–$34,999	1(7)
$35,000–$49,999	1(7)
>$50,000	1(7)
Marital status
Divorced	1(7)
Married	7(47)
Single	1(7)
Widowed	6(40)
Have health insurance
Insurance	12 (80)
Medicare	12 (80)
Additional insurance
Private	4(27)
Medicaid	1(7)
Self-reported general health
Excellent	(0)
Very good	(0)
Good	10 (67)
Fair	4(27)
Poor	1(7)
Diabetes treatmenta
Insulin	4(27)
Pills	14 (93)
Diet	4(27)
Exercise	1(7)
Comorbiditiesa	12 (80)
Hypertension	9(60)
Heart disease	5(33)
High cholesterol	5(33)

^aTotal is greater than 100%, as some report multiple types.

The mean score for the REALM was 41 (range 7–65). The mean score for the S-TOFHLA was 19 (range 0–33). These scores indicate marginal levels of health literacy, which imply difficulties with reading and comprehending written health materials.

HRQoL Questionnaires

All 15 subjects completed both sessions. Analyses of the 30 interviews yielded examples of three of the four major problem types: comprehension, bias/sensitivity, and response categories. Tables 5 and 6 provide examples of the most common problem types and specific issues identified for the ADDoL and DQOL-B, respectively. The tables are organized to report the findings by original question number, followed by examples of the issue and a suggested change for the selected items. Clearly, a next research step is for suggested changes to be reviewed by a panel of survey experts and undergo additional cognitive testing with a similar sample of African American elders.

TABLE 5.

Diabetes Dependent Quality of Life (ADDQoL) Survey: Key Issues and Suggested Revisions for Use With Older African Americans

Survey Item or Question Number From Survey Instrument	Key Issues Identified	Response to Probes	Suggested Actions
Q1. If you did not have diabetes, you would enjoy your leisure activities: Very much more Much more A little more The same Less	Comprehension (clarity, vague wording)	“What [do] you mean … ? I’m trying to figure out what you talking about … Maybe you need to look at and bring it down, make it more plain so I can understand exactly.”	Add clarification: By leisure activities I mean “what you would enjoy in your spare time, what you would do for fun.”
Q8. Do you, or would you like to, have a close personal relationship? Yes No	Comprehension (clarity, vague) Bias/sensitivity (sensitive wording; socially acceptable)	“No, I can’t say that, I got my wife. I thought you meant go out there and have somebody on the side, it put me in the mind of a bachelor”; “on the side, outside of marriage…. A close personal relationship is different than having a brother or sister, it’s like having a girlfriend or boyfriend.”	Add examples: (wife/husband, family member or friend) Q8. Do you have a close personal relationship with at least one person, such as a wife (husband), family member. or friend? A close personal relationship means that you have someone you feel close to and that understands you.
Q8a. If you did not have diabetes, your closest personal relationship would be: Very much better Much better A little better The same Worse		“Say that again … let’s see … the question don’t say with whom or what are you talking about with anybody or what? … But how that question is asked I didn’t know specifically what they meant. … They should ask specifically, ‘say do you a close personal relationship with X, Y, or Z’ … still don’t explain what kind of relationship. … That question could also make somebody think about an extramarital affair.”	Q8a. If you did not have diabetes, your closest family or friend relationship would be:
Q9a. If you did not have diabetes, your sex life would be: Very much better Much better A little better The same Worse	Response category (inappropriate assumptions, assumes current behavior; missing a doesn’t apply formatting or response)	“Because when you reach a certain age that thing goes away from you.” “No that is gone. … I’m 73 years old what I done did out there that’s it, now it’s time to work for the Lord see you ain’t doing nothing but fooling yourself, certain age you get that’s it once you hit 65 that’s it.” “I don’t know, because I have buddies and don’t have diabetes and they tell me about their misfortunate [sic].”	Add an optional category. Reframe lead in question. Some people feel diabetes affects their sex life. Does this apply to you? No (skip to Q10) Yes (if, yes then proceed to ask)
Q10a. If you did not have diabetes, your physical appearance would be: Very much better Much better A little better The same Worse	Comprehension (technical term)	“If I didn’t have diabetes I’d be able to do my housework.” “You talking about my health. … I think this is about my health, but not so sure … see right now I’m facing age … things I didn’t face before … I getting up in age and this is a new experience.”	Add clarification Q10a. If you did not have diabetes, your physical appearance (that is, the way you look) would be: Very much better Much better A little better The same Worse
Q11a. If you did not have diabetes, your self-confidence would be: Very much greater Much greater A little greater The same Less	Comprehension (clarity, technical term)	Self-confidence means “To keep things to myself and not talk to nobody cause I know it’s kept cause if I tell it to the next person then they’ll tell somebody else.”	Add clarification: Q11a. I will ask you a question about self-confidence. By self-confidence I mean “believing in yourself, believing that you can …” If you did not have diabetes, your self-confidence would be: Very much greater Much greater A little greater The same Less
Q12a. If you did not have diabetes, your motivation would be: Very much greater Much greater A little greater The same Less	Comprehension (clarity, technical term; vague)	“That one’s a little difficult for me. … I really don’t understand that one.” “I can’t think now. … How I’m doing. How I’m getting along … no that’s still not it … (sigh) … no I can’t get it.” “Some of the words I hate to say it, but I really don’t know them. A lot of those words I don’t know.”	Add clarification: Q12a. I will ask you a question about motivation. By motivation I mean “The will to do something,” “The inspiration …” “The drive …” If you did not have diabetes, your motivation would be: Very much greater Much greater A little greater The same Less
Q14a. If you did not have diabetes, your feelings about the future (e.g. worries, hopes) would be: Very much better Much better A little better The same Worse	Bias/sensitivity (sensitive wording; assumes constant behavior)	Well I think about it. … I try not to worry, but I think about it and trust in the Lord …” “To me it matters and it doesn’t matter cause I know at 68 I’m not going to live forever and you have to think about things like that … I don’t fret about it”	Modification: Replace the word “worry” Q14a. If you did not have diabetes, your feelings about the future, your concerns, and hopes would be: Very much better Much better A little better The same Worse
Q19a. If you did not have diabetes, your freedom to drink as you want (e.g., fruit juice, alcohol, sweetened hot and cold drinks) would be: Very much greater Much greater A little greater The same Less	Bias/sensitivity (inappropriate assumptions; double-barreled; sensitive wording)	“I don’t drink at all … no I don’t drink alcohol … never did” “Well number one, it would have to be water, tea, but alcohol would not be included.”	Modification: Remove the word “alcohol” Q19a. If you did not have diabetes, your freedom to drink as you want such as fruit juice or sweetened hot and cold drinks would be: Very much greater Much greater A little greater The same Less

TABLE 6.

Diabetes Quality of Life Brief Clinical Inventory (DQOL-B): Key Issues and Suggested Revisions for Use With Older African Americans

Survey Item or Question Number From Survey Instrument	Key Issues Identified	Response to Probes	Suggested Action
Survey response category Very satisfied Moderately satisfied Neither Moderately dissatisfied Very dissatisfied	Response category (technical term)	“What’s the meaning of moderately?” “I’m satisfied.” “I’m not satisfied.”	Modification: Replace “moderately” and “dissatisfied” with “somewhat” and “not satisfied,” respectively: Very satisfied Somewhat satisfied Neither Somewhat not satisfied Not at all satisfied
4. How often do you worry about whether you will miss work? Never Very seldom Sometimes Often All the time	Comprehension (clarity, vague; inappropriate assumptions)	“What type of work? Do you mean housework?” “That’s what I’m worried about because I can’t work anymore, I loved working . . . I miss the people I use to work with.” “I don’t hardly worry about it . . . because I don’t have that on my mind on missing work. . . . That question asking if I worry about missing work . . . my definition of missing work . . . I would be thinking about work.”	Reframe question: 4a. Some people feel diabetes affects whether you will be absent from your job. By job, I mean work that you do for pay. Do you have a job or do you work for pay? No (skip to Q5) Yes (If yes, go to Q4b) 4b. How often are you concerned about whether you will be absent from your job because of your diabetes? Never Not often Sometimes Often All the time
5. How satisfied are you with the time it takes to determine your sugar level? Very satisfied Moderately satisfied Neither Moderately dissatisfied Very dissatisfied	Comprehension (clarity, technical term) Response category	“I’m satisfied because my sugar never was too high . . . and I was able to keep it just about under control”; “I’m thinking about when I found out I had diabetes and how long it take me to bring [it] down. . . .” “. . . ’Cause sometimes when I check it it’s not what I expect it to be . . . when I check it and it be high I’m unhappy about it.”	Add a reframing “probe” statement prior to asking the question: Think about the time it takes to test your sugar level. Now, how satisfied are you with the time it takes to test your sugar level? Very satisfied Somewhat satisfied Neither Somewhat not satisfied Not at all satisfied
9. How often do you feel diabetes limits your career? Never Very seldom Sometimes Often All the time	Comprehension (clarity, technical term, vague; inappropriate assumptions, assumes constant behavior)	“Umm . . . career? That’s like what? (deep sigh) . . . I just don’t know . . . please repeat the question.” “Because I can’t walk like I want to walk” (probe: What is your definition of career?) “My career? Doing the things I want and like to do.” “How you mean, stuff I can do and do no more . . . work, sex, travel, church—get up to go to bathroom. . . . Make you go to the bathroom fast.”	Reframe question: For older adults consider more general notion of activities: 9. I will ask a question about activities. By activities I mean “what you do with your time from day to day.” How often do you feel diabetes limits your activities? Never Not often Sometimes Often All the time
10. How often do you have pain because of the treatment for your diabetes? Never Very seldom Sometimes Often All the time	Comprehension (clarity, technical term, vague)	“Pain . . . some people have pain in the knee in the arm. I don’t hardly have no pain . . . I’ve never been to the doctor really for pain.” “Well pain from being a diabetic I only have pain sometimes in my legs and sometimes in my head.”	Add clarification: By treatment I mean, “needles for insulin or finger sticks for testing your blood sugar”: Never Not often Sometimes Often All the time
11. How satisfied are you with the burden your diabetes is placing on your family? Very satisfied Moderately satisfied Neither Moderately dissatisfied Very dissatisfied	Bias/sensitivity (inappropriate assumptions; sensitive wording, burden has negative connotation) Response category (technical term)	“It doesn’t bother my family so I should say never . . . it’s not a burden to them” [grimacing at question]. . . . That question to me is saying something like are they pleased that I have diabetes and I don’t feel they are pleased or displeased with it . . . there’s no way it’s affecting them . . . no more than they have to go to store and get my medicine. . . .”	Reframe question: 11a. Some people feel their diabetes places a burden on other family members. Does this apply to you? No (skip to Q 12) Yes (then ask question) How satisfied are you with the burden your diabetes is placing on your family? Very satisfied Somewhat satisfied Neither Somewhat not satisfied Not at all satisfied
13. How often do you worry about whether you will pass out? Never Very seldom Sometimes Often All the time	Bias/sensitivity (sensitive wording; assumes constant behavior)	“I never worry about passing out . . . faith, I have the faith that I will never pass out.” “No I don’t worry about passing out and I’m praying that I don’t pass out . . . no I don’t worry about that”	Reframe question: How often are you concerned about passing out because of your diabetes? Never Not often Sometimes Often All the time

Most observed problems were coded as comprehension/communication (e.g., clarity and vague wording). The respondents did not understand the questions because of unfamiliar language and uncertainty about the meanings of specific words. Respondents reported some questions to be confusing and were unsure as to what was being asked. Many of the unknown or variable interpretations of terms and phrases surfaced with verbal probing. To a lesser, albeit important, extent, bias/sensitivity issues (i.e., judgment process) may reflect social and contextual variables that influenced the survey response process. Key examples were the words “worry” and “alcohol.” The word “worry” appeared to have a negative connotation and in some instances posed a spiritual conflict; in other words, to “admit” to worrying equated to a lack of faith. “Alcohol,” on the other hand, was listed as one example among other drinks, but respondents focused mainly on alcoholic drinks, emphasizing that they did not drink alcohol (see questions 14a and 19a in Table 5).

Comparison of ADDQoL and DQOL-B

The findings suggest that this cohort of participants required more mental processing (an indication of respondent burden) for the ADDQoL instrument than for the DQOL-B, with respondents having to consciously reframe their realities and think about what certain aspects of life would be like without diabetes. In addition, 4 of 19 questions had missing answers in the ADDQoL survey; again, this may be due to unfamiliar terms or limited response categories. These issues were not observed with the DQOL-B. Both instruments, however, presented challenges for this high-risk group. Tables 5 and 6 show that 8 of 19 (42%) ADDQoL questions and 7 of 15 (47%) DQOL questions presented challenges to some of the respondents. Additionally, more respondents indicated difficulty understanding the ADDQoL survey instructions.

This study purposely recruited elders with low education levels. Table 7 reports the literacy scores and selected demographic characteristics of the participants with the most problems of processing on both instruments. It also provides information on how the scores should be interpreted. Challenged participants were older, had less than a high school education, and had been living with diabetes for many years. Clearly, these elders struggled with or could not answer definitively or validly because of unfamiliar language and troublesome response categories such as “moderately” used with the DQOL Brief Clinical Inventory.

DISCUSSION

The study reveals that well-accepted and utilized instruments for studies of persons with diabetes presented challenges for this group of older African Americans. This presents major measurement concerns about their reliability and validity, critically important concerns when assessing needs or evaluating interventions. Consistent with Willis (2005), this study identified several areas of concern: (a) use of unfamiliar language, (b) use of words without universal meaning for the population being studied, (c) contextual differences that may be culturally based, and (d) answer categories that do not fit internally generated response categories. Thus, the domains of comprehension/communication, bias/sensitivity, and response category continue to present challenges for survey designers and the intended respondents.

The researchers found that this purposive sample of older respondents did not distinguish between certain functional limitations and/or level of satisfaction as being directly identified with their diabetes. Although the survey instruments used in this study were related to diabetes’ impact on QoL, respondent’s perception of aging or one’s age was apparently incorporated into the processing of their responses to items. Respondents did not make a clear distinction that problems were related to diabetes but frequently referred to their age as a possible contributor to stressors they encountered and subsequently their ratings of QoL. On occasion, there were responses such as “I don’t know because I’ve never been this age before” or, as another respondent said, “I think about my age and my diabetes … the things I don’t do no more, I can’t do no more and is it because of my age or diabetes.” Furthermore, the inability of respondents to delineate certain characteristics to their disease process versus normal aging and the impact on QoL is worth noting. Adequate pretesting of the instrument with the population being studied is essential to help the researchers identify the previously specified issues with instrumentation.

This study confirms that the cognitive interview method is useful for evaluating questions so that they can be modified for use with older African Americans. By clarifying the meaning and improving comprehension of questions’ intent and/or meaning, survey errors related to misinterpretation are potentially avoided. Moreover, recognizing potential cross-cultural variations led to identifying survey questions that do not provide a “fit.” Probes such as “Tell me in your own words what the question is asking” seek paraphrasing and provide rich information that can lead to alternative word choices. These qualitative data also confirm that problems were not always purely cognitive but at times were logical interpretations from the patient’s perspectives and life experience (Willis, 2005). Other researchers (Gerber, 1999; Loomis & Rothgeb, 2005) have also conceptualized an ethnographic approach to cognitive interviewing to determine whether a particular question is commonsensical ( provides consistency across the cultural spectrum). Previous studies (Vileikyte et al., 2003) may have excluded a significant number of older respondents, those less educated and living with more comorbid chronic conditions. Our results suggest that valuable data exist within these groups, and the option of interviewer-administered surveys to reduce respondent burden is feasible. The application of cognitive principles and techniques has been described as inconsistent and not well documented (Drennan, 2003; Willis, Reeve, & Barofsky, 2005). Calls for use of an analysis framework (Knafl et al., 2007) and taxonomies of problem classification are recognized (Drennan, 2003; Willis et al., 2005). This study, though small, provides a needed systematic approach to investigating respondents’ cognitive processing of an important construct closely associated with chronic illness.

There is no one perfect or all-inclusive HRQoL instrument to measure the variable dimensions included in the ADDQoL and DQOL Brief Clinical Inventory (see Table 1). Consequently, researchers and clinicians are advised to review survey content carefully when conceptualizing their research studies and/or evaluating intervention strategies.

The ADDQoL presents hypothetical constructed questions that prior research has shown can lead to failures to answer a question or misrepresentations of the respondents’ thoughts (Willis, 2005). We did not identify this as a problem in our study sample. Although there is more respondent burden with the ADDQoL, respondents rather quickly began to employ a pattern necessary for them to encode and generate responses. Prior to offering a response, respondents consistently restated, “If I did not have diabetes.” This research also extends other evidence that aging does influence response shifts, and, therefore, question modification may be necessary to eliminate age-based frame of reference effects (Willis et al., 2005).

The REALM and S-TOFHLA have been widely used to measure patients’ health literacy skills. In this study, several observations may suggest the need for other methods to assess health literacy. For example, participants added the letter “s” to the end of words, and this was a common mispronunciation observed with the REALM. In the region where the participants reside, some “locals” routinely add and/or drop consonants. This might have an impact on health communication, particularly when attempting to repeat instructions related to medication adherence. The S-TOFHLA is a timed assessment, and 11 (73%) participants were unable to complete it within the allotted time. It was also observed that some respondents worked at a “careful” pace, and others had difficulty manipulating the pages of a self-administered literacy test. Although these observations are anecdotal, they deserve systematic consideration prior to deciding on the usefulness of a health literacy instrument for certain groups. Furthermore, novel ways to assess health literacy in clinical settings have emerged (Weiss et al., 2005) and will require further study. Notably, the field of health literacy research is experiencing tremendous growth and recognition as an important link to self-management, although research associating health literacy scores and health outcomes is less clear. Overall, these areas provide substantial opportunities for research, evaluation, and intervention work.

Limitations of this study and methodology should be considered when interpreting the findings. Clearly, this investigation has a small sample size, and findings of this pilot study should be confirmed with an additional sample of elders and others; however, the purpose of cognitive interviewing is not statistical estimation. With this purposive, nonrandom sample, we were successful in recruiting low-literacy adults, a group particularly difficult to engage in research. In addition, the two instruments selected for study do not represent all the conceptual domains that may be important to a person’s own view of their QoL. The physical and social environment may not fully replicate the survey field interview process.

In conclusion, the paucity of research on HRQoL among older African Americans presents a challenge and an opportunity for further investigation. HRQoL research challenges continue to be the refinement of conceptual frameworks, refinement and validation of measures, and application of findings to grow the much-needed evidence base relating to interventions and outcomes. This particular study, by considering the perspective of a high-risk group such as older African Americans with low literacy, has added to a rather limited body of knowledge. By combining probing approaches and an analytic typology of issues (Willis, 2005), the researcher was able to extend understanding of the question-and-response process, thereby providing a richer description related to comprehension difficulties, questions deemed not culturally relevant (cultural relevancy), and grounding of the phenomena. This study has implications for further research on the appropriateness of established and new instruments. Cognitive interviewing is resource intensive, requiring time and specialized skill as well as added cost to a project. Yet ignoring this investment may result in inappropriate interpretations about the effectiveness of much-needed interventions to improve care and outcomes and to reduce disparities.