Refining and supplementing candidate measures of psychological well-being for the NIH PROMIS®: qualitative results from a mixed cancer sample

Abstract

Purpose

The Patient-Reported Outcomes Measurement Information System (PROMIS) is a National Institutes of Health initiative designed to improve patient-reported outcomes using state-of-the-art psychometric methods. The aim of this study is to describe qualitative efforts to identify and refine items from psychological well-being subdomains for future testing, psychometric evaluation, and inclusion within PROMIS.

Method

Seventy-two items from eight existing measures of positive affect, life satisfaction, meaning & purpose, and general self-efficacy were reviewed, and 48 new items were identified or written where content was lacking. Cognitive interviews were conducted in patients with cancer (n = 20; 5 interviews per item) to evaluate comprehensibility, clarity, and response options of candidate items.

Results

A Lexile analysis confirmed that all items were written at the sixth grade reading level or below. A majority of patients demonstrated good understanding and logic for all items; however, nine items were identified as “moderately difficult” or “difficult” to answer. Patients reported a strong preference for confidence versus frequency response options for general self-efficacy items.

Conclusions

Altogether, 108 items were sufficiently comprehensible and clear (34 positive affect, 10 life satisfaction, 44 meaning & purpose, 20 general self-efficacy). Future research will examine the psychometric properties of the proposed item banks for further refinement and validation as PROMIS measures.

Introduction

The Patient-Reported Outcomes Measurement Information System (PROMIS®; http://nihpromis.org) is an NIH Roadmap initiative designed to improve patient-reported outcomes using state-of-the-art psychometric methods [1, 2]. The main goal of PROMIS® is to develop and evaluate a set of publicly available, efficient and flexible measurements of patient-reported outcomes for use by clinicians and patients in diverse research and clinical settings [1]. Despite the conceptual breadth of PROMIS®, in the initial wave of instrument development efforts, measures of psychological well-being for adults with acute and chronic health conditions were not included in the measurement framework. Many patient-reported measures of health status (e.g., pain, fatigue, depression) are conceptualized as a lack of symptoms rather than the presence of well-being. Thus, development of PROMIS® item banks for psychological well-being will address an important gap in the measurement framework and allow for precise measurement of emotional health rather than merely the absence of symptoms.

Informed by models of psychological well-being [3–11], we identified four cross-cutting subdomains: (1) positive affect—feelings that reflect a level of pleasurable engagement with the environment such as happiness, joy, excitement, enthusiasm, and contentment [12]; (2) life satisfaction—a person’s cognitive evaluation of life experiences and whether s/he likes her/his life or not [13]; (3) meaning and purpose—the extent to which a person feels her/his life matters or makes sense [14]; (4) general self-efficacy—a person’s belief in her/his capacity to manage functioning and have control over meaningful events [15]. Consensus on these subdomains was sought through a modified Delphi process and guided by a review of the literature, feedback from experts in the area of psychological well-being, follow-up semi-structured interviews with a subset of these experts, and discussion within the project team and among content expert consultants [16]. Importantly, each of these well-being subdomains represents key indicators of positive emotional health and has important linkages to other health outcomes [13, 14, 16]. This qualitative study aimed to identify and refine items from these related but distinct psychological well-being subdomains with a mixed cancer sample for future testing, psychometric evaluation, and inclusion within PROMIS®. Patients with a history of cancer provide an ideal sample for exploring psychological well-being themes since cancer can be a catalyst for reflection, psychosocial growth, and meaning [17, 18].

Methods

Participants and procedures

Items from existing measures of positive affect, life satisfaction, meaning & purpose, and general self-efficacy [16, 18] were reviewed for reading level, clarity, simplicity, and translatability. Items were excluded if they were > 6th grade reading level, double-barreled, colloquial or idiomatic, or had intellectual property restrictions. Translatability review was used to identify potential conceptual or linguistic difficulties in items and to suggest alternate wording more suitable for a culturally diverse population. In cases where content was lacking (e.g., missing or under-represented component(s) of a psychological well-being subdomain), new items were written by study investigators (JS, DC) and content experts (CP, LG, MFS, TM) to ensure adequate breadth of the well-being subdomains. Any inconsistencies in investigator recommendations were resolved through consensus. To minimize respondent burden, response options were standardized to a limited set of options. Four study forms of 30 items each were created from the 120 candidate items. All forms included items from each of the four psychological well-being subdomains.

Study procedures were approved by the Northwestern University Institutional Review Board and eligible participants were identified via electronic medical record review and approached in-clinic at the Robert H. Lurie Comprehensive Cancer Center or contacted by phone after approval was obtained from patients’ providers. Eligibility criteria included the following: (1) able to read and understand English, (2) able to provide informed consent, (3) at least 18 years of age, (4) currently or previously diagnosed with breast, colorectal, lung, or prostate cancer (the four most common cancer types among adults), and (5) a life expectancy of at least 6 months. Interested participants were consented and completed cognitive interviews in-person at a private office suite at the Northwestern University Feinberg School of Medicine or by phone. Interview guides (and probes) were adapted from existing interview guides used in other PROMIS® and similar measure development work [19, 20]. Consistent with PROMIS® guidelines for the cognitive interview phase of item development [21], we recruited a purposive sample so that one patient in each group of five had limited educational attainment (i.e., high school or below), and two patients in each group of five were racial or ethnic minorities. We also sought to balance group assignment to interview study form by gender, treatment status (on vs. off), cancer type, and cancer stage. Upon completion of the interviews, participants were compensated $30 for their time.

Study measures

Participants reviewed items from the NIH Toolbox’s Positive Affect, General Life Satisfaction, Meaning & Purpose, and General Self-Efficacy Item Banks [16, 22]. The Positive Affect Item Bank included 34 items previously adapted from among the PANAS-X [23], Affectometer 2 [24], and the FACIT-Sp [25]. The Life Satisfaction Item Bank included 10 items previously adapted from the Satisfaction with Life Scale [13] and Students’ Life Satisfaction Scale [26]. The General Self-Efficacy Item Bank included items previously adapted from the General Self-Efficacy Scale [15] with parallel item content (10 items each) for both frequency (“never” to “very often”) and newly written confidence response options (“I am not at all confident” to “I am very confident”). The Meaning & Purpose Item Bank included 18 items previously adapted from the Life Engagement Test [27], Meaning in Life Questionnaire-X [14], and the FACIT-Sp [25] and also included 38 newly written items to develop a more robust meaning and purpose item bank for the NIH PROMIS®.

Analysis

Each of the 120 candidate items was reviewed by five patients (30 items per patient) to evaluate comprehensibility (“Can you say this question in your own words?” “How did you choose your answer?”), clarity (“Was this question easy or hard to answer?” “Can you think of an easier way to word this question?”), and preference for response options (“How easy is it to tell the difference between each response group?” “Which group of responses is easiest/hardest to understand?” “Which group of responses do you prefer and why?”). Participant responses were coded by the study coordinator (MAS) for understanding and logic (1 = “understanding and/or logic is poor/different/wrong” to 3 = “understanding and/or logic is full/good”) and for ease of answering (1 = “difficult to answer” to 3 = “easy to answer”). Coding decisions were reviewed, discussed, and modified, as needed, by the study principal investigator (JS) and co-investigator (DC). Participant preferences for response options were summarized as percentages. Measurement science (JS, EH, DC) and content experts (CP, LG, MFS, TM) reviewed cognitive interview results and provided recommendations for reducing redundancy, maximizing clarity, and enhancing conceptual breadth.

Results

During the translatability review, ten items were identified as potentially problematic and were re-written prior to cognitive interviewing to be less idiomatic or ambiguous. A Lexile® analysis of candidate items found that all items were written at the sixth grade reading level or below. Twenty patients (M = 62.0 years old, SD = 10.8) completed cognitive interviews. Additional sociodemographic and clinical characteristics are available in Table 1.

Table 1.

Participant demographic and clinical characteristics

	N=20
	N	%
Gender
Female	10	50
Male	10	50
Ethnicity
Hispanic origin	1	5
Non-hispanic origin	19	95
Race
White	12	60
Black/African America	7	35
Other	1	5
Education
High school degree/GED or less	6	30
Some college	6	30
College degree	5	25
Graduate degree	3	15
Tract income level
Low (median family income % is < 50%)	3	15
Moderate (median family income % is > = 50% and < 80%)	6	30
Middle (median family income % is > = 80% and < 120%)	4	20
Upper (median family income % is > = 120%)	7	35
Cancer type
Breast	5	25
Prostate	5	25
Colorectal	5	25
Lung	5	25
Cancer stage
Early (Stages 0-II)	9	45
Advanced (Stages III-IV)	11	55
Treatment status
On treatment	10	50
Off treatment	10	50

A majority of patients (at least 3 out of 5 in every “set”) indicated good understanding and logic for all candidate items (See Table 2 for sample responses). However, nine (6 meaning & purpose) items were identified by a majority of patients as “moderately difficult” or “difficult” to answer (e.g., “I realize my life has a central theme”). In terms of general self-efficacy response options, patients reported a preference for confidence (55%) vs. frequency (30%) options.

Table 2.

Sample cognitive interview responses

Content	Item stem	Response options	Patients’ comments about items (patients’ Age/Sex)	Example of…
Positive affect	I felt attentive	Not at all A little bit Somewhat Quite a bit Very much	“Same as question 22 (I felt cheerful). I have a wonderful life. I’m blessed to be here. I try to be cheerful and happy.” (69/M)	Poor understanding and logic
Life satisfaction	In most ways, my life is close to perfect	Strongly disagree Disagree Slightly disagree Neither agree nor disagree Slightly agree Agree Strongly agree	“Hard because of the word ‘perfect’. I don’t know how to answer it. What is perfection?” (59/F)	Difficult to answer
Meaning & purpose	I believe there is an ultimate meaning of life	Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree	“What exactly is an ‘ultimate meaning of life’? I have ethical and moral standards, but I don’t know if that’s an ultimate meaning. This one was very hard; I had no answer for it.” (70/M)	Partial understanding; difficult to answer
Self-efficacy	I can handle whatever comes my way	Never Almost Never Sometimes Fairly Often Very Often	“I have a circle of people to help me. I can think of cancer being the worst thing that came my way. I rely on people.” (81/F)	Full understanding and good logic

Based on cognitive interview data and expert review, 12 items were omitted from further consideration (all from the meaning & purpose item pool). Five of those items were excluded as a result of cognitive interview feedback which revealed that patients had poor understanding, incomplete logic, and/or difficulty answering the items. The remaining 7 items were excluded after expert review which highlighted a lack of clarity or simplicity (2 items), redundancy with other items (3 items), and content that was beyond the scope of the proposed construct (2 items). In addition, 3 items that were considered “moderately difficult” or “difficult” to answer (2 general self-efficacy items and 1 life satisfaction item) were retained for further testing given their inclusion in existing legacy measures [13, 15]. Figure 1 provides a summary of the development and refinement of the PROMIS® item banks for psychological well-being based on the qualitative review process.

Fig. 1.

Exclusion considerations and results

Conclusions

Altogether, 108 items were identified for the next phase of testing. These included 34 positive affect, 10 general life satisfaction, 18 meaning and purpose, and 10 general self-efficacy items from the NIH Toolbox. Within this process, confidence response options were written for the 10 general self-efficacy items to better reflect patient preferences and align with self-efficacy theory [28]. For the meaning and purpose items, additional content was identified and written to enhance conceptual breadth of this important subdomain of psychological well-being.

As a result of the qualitative review process, the item pool for psychological well-being was refined in preparation for quantitative testing. Importantly, all items were sufficiently comprehensible and free of ambiguity. Perhaps not surprisingly and relative to other psychological well-being items, meaning and purpose items were more difficult to answer. At the construct level, meaning and purpose can sometimes be nebulous with ambiguous conceptual boundaries, resulting in challenges to identify clear and comprehensible items that adequately reflect this important construct [29]. This resulted in “pruning” of the meaning and purpose item pool to identify the optimal candidates for additional testing. Future research will examine the psychometric properties of the proposed psychological well-being item sets using a general population sample for further refinement and validation as potential PROMIS® measures.