Development of an item bank for computerized adaptive testing of self-reported cognitive difficulty in cancer patients

Abstract

Background

The European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life Group is developing computerized adaptive testing (CAT) versions of each scale of the EORTC Quality of Life Questionnaire (EORTC QLQ-C30). This study aims to develop an item bank for the EORTC QLQ-C30 cognitive functioning scale, which can be used for CAT.

Methods

The complete developmental approach comprised four phases: (I) conceptualization and literature search, (II) operationalization, (III) pretesting, and (IV) field-testing. This paper describes phases I–III.

I) A literature search was performed to identify self-report instruments and items measuring cognitive complaints on concentration and memory. II) A multistep item-selection procedure was applied to select and generate items that were relevant and compatible with the ‘QLQ-C30 item style.’ III) Cancer patients from different countries evaluated the item list for wording (ie, whether items were difficult, confusing, annoying, upsetting or intrusive), and whether relevant issues were missing.

Results

A list of 439 items was generated by the literature search. In the multistep item-selection procedure, these items were evaluated for relevance, redundancy, clarity, and response format, resulting in an list of 45 items. A total of 32 patients evaluated this item list in the pretesting phase, resulting in a preliminary list of 44 items.

Conclusion

Phase I–III resulted in an item list of 44 items measuring self-reported cognitive complaints that was endorsed by international experts and cancer patients in several countries. This list will be evaluated for its psychometric characteristics in phase IV.

Cancer patients may experience cancer-related cognitive impairments at various stages of their disease trajectory. These cognitive impairments may include impaired verbal and visual memory, problems with attention/concentration, impaired language skills, slower information processing speed or difficulty in executive functioning, among other cognitive disorders.^1–3 Cognitive impairments may arise in patients with malignancies outside the central nervous system (CNS), as well as in patients with brain tumors, due to the impact of the disease itself or its treatment, possibly aggravated by psychological distress, fatigue, and depression.⁴ In patients with non-CNS malignancies, these impairments are related mainly to the neurotoxic effects of systemic chemotherapy, biologic response modifiers, and adjuvant hormonal therapy that may penetrate the healthy brain tissue.^5–7 Patients with systemic cancer with a predisposition to metastasize to the brain can be treated prophylactically with cranial irradiation, which poses an additional risk for cognitive dysfunction.⁸ Compared with patients with non-CNS malignancies, the prevalence of cognitive impairments is higher in patients with brain metastases and primary brain tumors.⁹ A brain tumor itself may cause cognitive deficits by invasion of the healthy brain tissue or by compression caused by edema surrounding the tumor.^10–13 Moreover, antitumor treatment including surgery, chemotherapy, targeted therapy, and (whole brain) radiotherapy, as well as use of antiepileptic drugs or corticosteroids, may induce or further aggravate cognitive impairments in brain tumor patients.^{11,12,14–17}

Neurocognitive functioning is objectively assessed with standardized tests.¹⁸ Moreover, self-reported instruments exist to assess cognitive complaints.^19,20 Agreement between objective performance-based cognitive tests and subjective self-report cognitive complaints is poor.^9,21 Several self-report questionnaires on health-related quality of life (HRQOL), covering physical, psychological, and social functioning, also include items on cognitive complaints.^22–25 However, a limitation of these self-report measures is the lack of measurement precision, particularly at the level of the individual patient. With traditional questionnaires all patients have to answer the same set of items for results to be comparable. With this approach, a balance has to be found between optimizing measurement precision (requiring longer questionnaires) and limiting response burden (requiring shorter questionnaires).

One approach to overcome these limitations is computerized adaptive testing (CAT), which is based on a statistical method called item response theory (IRT).²⁶ Based on an algorithm (developed in a large sample of patients), CAT selects those items from an item bank that are most informative for characterizing the complaints of an individual patient. This means that the computer program selects a new item based on the response to previous items, aiming to maximize the information obtained. In general, with IRT-CAT, fewer items are needed to obtain precise measurement, thereby reducing the response burden for the individual patient. An advantage of IRT is that even though patients answer different subsets of items, scores are directly comparable. Thus, CAT measurements tailor the item set to the individual patient, thereby increasing the measurement precision and reducing the response burden, without loss of comparability between subjects. An ongoing project developing CAT for patient-reported outcomes is the Patient Reported Outcomes Measurement Information System (PROMIS) project in the United States.²⁷

Similarly, the European Organisation for Research and Treatment of Cancer Quality of Life Group (EORTC QLG) is developing CAT versions of all HRQOL scales of the EORTC Quality of Life Questionnaire (EORTC QLQ-C30), except overall health/quality of life. Although the QLQ-C30 is a generic questionnaire measuring HRQOL in all cancer patients, the cognitive functioning scale may be particularly relevant for brain tumor patients as well as patients with systemic cancer with a predisposition to metastasize to the brain or those vulnerable to treatment-related cognitive impairments. The aim of this study was to develop an item bank for the EORTC QLQ-C30 (cognitive functioning scale), focusing on items related to concentration and memory, which can be used for CAT.

Methods

The development of the CAT item bank was based on the strategy described by Petersen et al.²⁸ In short, there are 4 developmental phases: (I) literature search, (II) operationalization, (III) pretesting, and (IV) field-testing. This article describes the first 3 phases and provides a definition and conceptualization of the concept cognitive complaints, on which the item bank will be based.

Definition and Conceptualization of Cognitive Complaints

To ensure comparability between the CAT instrument and the original EORTC QLQ-C30 questionnaire, the CAT Working Group decided that the additionally developed items for the CAT instrument should reflect the same underlying concept as the 2 original cognitive complaints items of the EORTC QLQ-C30. Moreover, these items should differ in level of severity, to more accurately estimate cognitive complaints with CAT. Cognitive functioning has been defined as ‘a process by which sensory input is elaborated, transformed, reduced, stored, recovered, and used’²⁹ and more specifically as ‘an intellectual process by which one becomes aware of, perceives, or comprehends ideas. It involves all aspects of perception, thinking, reasoning, and remembering’.³⁰ Cognitive functioning is seen as a multidimensional concept, comprising several domains.³¹

The 2 original QLQ-C30 cognitive functioning items will also be included in the item bank: (Q20) ‘Have you had difficulty in concentrating on things, like reading a newspaper or watching television?’ and (Q25) ‘Have you had difficulty remembering things?’ Domains that were considered to be relevant for Q20 were concentration and attention. Although concentration and attention reflect 2 distinct domains in the literature, the definitions refer to the same theme. We therefore decided to combine concentration and attention. The domain memory was considered to be relevant for Q25. Therefore, the items selected for the cognitive functioning item bank were classified as being related to “concentration (including attention)” or “memory.” Moreover, new items should be rated on the same 4-point Likert-type scale as the original items, ranging from ‘not at all’ to ‘very much’ and refer to the same timeframe: ‘during the past week.’

Phase I: Literature Search

We performed a systematic literature search in the e-resources PROQOLID, PubMed, PsycINFO and the EORTC QLG Item Bank until June 2009. The aim of this search was to identify self-report instruments and items measuring cognitive complaints, which may serve as a basis for the generation of new items. The search strategy in PubMed/PsycINFO consisted of a combination of 3 search strings: 1 related to cognitive complaints, 1 related to assessment, and 1 related to cancer. For the other databases, we used keywords related to cognitive complaints. The full search strings/keywords are outlined in Supplementary Table 1. We screened all retrieved abstracts for potential relevant items. For each item, we extracted the item text and information on the original instrument and response format.

Phase II: Operationalization

In each step of this phase the item list was independently reviewed by at least 2 reviewers for relevance, redundancy, clarity, and response format. The complete item selection procedure consists of 6 steps. After each step, results were compared and discrepancies between reviewers were resolved in consensus.

• Step 1. Item classification

We classified items into one of the aspects of cognitive complaints assessed by the QLQ-C30 or into an ‘other’ category. Only items compatible with the EORTC conceptualization of cognitive complaints in the EORTC QLQ-C30 were retained in the item list.

• Step 2. Item deletion

Based on consensus between raters, we removed redundant items (ie, items similar in wording and meaning to another item) from the item list, as well as items that were incompatible with the QLQ-C30 style (ie, response categories and timeframe). We compared each set of duplicate items for clarity and similarity to the QLQ-C30 item style, and retained items that were most compatible with that style. We deleted items that could not be reformulated into the EORTC style.

• Step 3. Item formulation

Based on the content and formulation of the items retained in step 2, we formulated new items fitting the QLQ-C30 item style. In step 3.1 (item formulation), each reviewer independently formulated new items fitting the response categories and the timeframe of the QLQ-C30 items. Next, a consensus list of items was made based on the individual item list. Similarly to step 2, redundant items were deleted from the item list. In step 3.2 (review of the new items), 2 experts in the field of cognitive functioning (EJJH and MJBT) reviewed the candidate items for wording and relevance. Again, items were potentially revised or deleted in case of redundancy and/or inappropriateness.

• Step 4. Rating of the items

To assess if the current set of items covered the continuum of cognitive complaints, items were categorized (based on expert opinion) as reflecting patients with poor, moderate, or good level of cognitive functioning. Disagreement between raters was solved in consensus.

• Step 5. Generation of new items

In case of insufficient coverage of the measurement continuum, and when possible, new items were generated.

• Step 6. Expert evaluations

We applied a cross-cultural evaluation approach with international experts to maximize broad applicability of this CAT version of the EORTC QLQ-C30 cognitive functioning scale. Three groups of experts evaluated the items for their relevance to the construct of cognitive complaints, their appropriateness, and whether they were clear and well defined. Moreover, the experts were requested to suggest items that were currently lacking. First, 2 members of the EORTC QLG (NKA and JCR; experts in the field of neuro-cognition and HRQOL) reviewed the conceptualization of the concept cognitive complaints as well as the item selection procedure (step 6.1, internal peer review). Subsequently, the list of candidate items resulting from step 6.1 was circulated to 2 members of the CAT-group (experts in the field of questionnaire development) for evaluation (step 6.2, review CAT-group). Finally, 10 international experts in the field of HRQoL questionnaire development and neuropsychology (external to the EORTC CAT-group) were asked to evaluate the candidate item list (step 6.3, review international experts). After each evaluation step, items could be revised or deleted, if required.

Phases I and II were carried out in English. The English language items were then translated into Danish and French for phase III. Translation was performed by the Translation Office of the EORTC and followed well-established guidelines, including back and forward translations.³²

Phase III: Pretesting

To determine the appropriateness of the selected items for the target population and to ensure content validity, we pretested the preliminary item list in a sample of cancer patients recruited from different countries. Pretesting consisted of administering the item list, followed by a structured interview. In the interview, patients were asked to evaluate each item for wording (ie, whether the items were difficult, confusing, annoying, upsetting, or intrusive), and whether relevant issues were missing. Any other comments about the items were also recorded.

Ethical committees of all participating centers approved the study and patients gave their written informed consent.

Results

A summary of all developmental steps is provided in Figure 1.

Fig. 1.

Flowchart developmental steps for the cognitive complaint item bank. CF, cognitive functioning.

Phase I: Literature Search

The literature search revealed 44 instruments including 439 items related to cognitive complaints.

Phase II: Operationalization

• Step 1. Item classification

Each of the 439 items was classified as measuring concentration (n = 64), memory (n = 229), both concentration and attention (n = 1), or another cognitive domain (n = 145). Only items measuring concentration (including attention) and memory were retained. Reviewers agreed on item classification in 418/439 (95%) cases; the 21 disagreements were resolved by consensus. At this stage, the item list included 294 items.

• Step 2. Item deletion

The reviewers agreed in 226/294 (77%) cases on whether items should be retained or excluded, based on redundancy or incompatibility with the QLQ-C30 response format. The 68 disagreements were resolved by consensus. A total of 240 items (82%) were deleted in this step, 178 (74%) of which were deleted because of redundancy. After this step the item list included 54 items.

• Step 3. Item formulation

  • Step 3.1 Item formulation.

The 54 remaining items were used as basis for formulating new items fitting the QLQ-C30 item style. The 2 reviewers agreed upon a consensus list consisting of a total of 47 items fitting the QLQ-C30 style.

• Step 3.2 Review of the new items.

After this first reformulation, the 47-item list was reviewed by 2 experts in the field of cognition. In this step, 4 items were deleted and 4 items were revised. This resulted in an item list containing 43 items.

• Step 4. Rating of the items

All 43 candidate items were assessed for their level of severity of cognitive complaints. In one-third of the cases, raters disagreed on the most appropriate category, which was resolved in consensus. This resulted in 13 items classified as most relevant for patients with poor cognitive functioning, 18 items for patients with moderate cognitive functioning, and 12 items for patients with good cognitive functioning. Although the current set of items covers the continuum of cognitive functioning, only 9 of the items covered the domain concentration and only 1 of these items was rated as being relevant primarily for patients with poor cognitive functioning.

• Step 5. Generation of new items

Although the item list at this point contained only one item about concentration that is most relevant for patients with poor CF, the experts did not envisage additional items at this stage, and decided to await feedback from the other experts.

• Step 6. Expert evaluations

  • Step 6.1 Internal peer review.

Two members of the EORTC QLG reviewed and agreed on the conceptualization of the concept cognitive functioning. Based on their review of the item selection procedure, 1 item was revised and 1 item was deleted, resulting in 42 items. Next, the rating of 5 items was changed, resulting in 14 items for poor cognitive functioning, 20 items for moderate cognitive functioning, and 8 items for good cognitive functioning.

• Step 6.2 Review CAT-group.

The second step in the review procedure included revisions by members of the EORTC CAT-group. Comments from members resulted in revision of 3 items (clarification of the items). Hence, the item list still consisted of 42 items.

• Step 6.3 Review by international experts.

All 42 remaining items were evaluated by 10 international experts from Denmark (n = 4), the Netherlands (n = 3), the United States (n = 2), and Germany (n = 1) for their relevance, their appropriateness, and for whether they were clear and well defined. Items for which 2 or more experts had made a comment were considered for revision/deletion. This evaluation resulted in a total of 5 items that were deleted, 10 items that were revised, and 6 items that were newly generated (including 1 item for concentration that could be rated as being particularly relevant for patients with poor cognitive functioning). Thus, after the expert evaluations, the preliminary item list consisted of 43 items plus the 2 original items of the QLQ-C30 CF scale. These 45 items were then evaluated in the pretesting phase.

Phase III: Pretesting

A total of 32 patients were interviewed in the pretesting phase. Patients were from Denmark (38%), France (31%), and the United Kingdom (31%) and varied in tumor type and stage, as well as treatment. Details on patient characteristics are shown in Table 1. In general, the patients had very few comments about the items. None of the items were rated as difficult, confusing, annoying, upsetting, or intrusive by more than 2 patients. One item was deleted because of ambiguity and 2 items were revised, 1 because of ambiguity and the other because of lack of clarity. No new items were generated by patients.

Table 1.

Clinical characteristics of the 32 patients participating in the pretesting phase

Characteristic		N (%) / mean
Gender	Male	16 (50%)
	Female	16 (50%)
Country	Denmark	12 (38%)
	France	10 (31%)
	United Kingdom	10 (31%)
Age in years, mean (range)		61 (35–84)
Cancer stage	I–II	12 (38%)
	III–IV	18 (56%)
	Unknown	2 (6%)
Diagnosis	Breast	5 (16%)
	Gastrointestinal	11 (34%)
	Head & Neck	6 (19%)
	Gynecological	7 (22%)
	Other	3 (9%)
Current treatment	Chemotherapy	16 (50%)
	No current treatment	12 (38%)
	Other	4 (12%)
	Unknown	0 (0%)

After phase III the item list consisted of 44 items (including the 2 items from the QLQ-C30). This preliminary item bank for cognitive complaints will be field-tested in phase IV. See Table 2 for the complete item list.

Table 2.

Item list for field-testing in phase IV

Item number	Item text
Item 01**	Have you had difficulty performing two tasks simultaneously, eg, having a conversation while cooking?
Item 02*	Have you had difficulty remembering words, eg, a word was “on the tip of your tongue” but you could not quite find it?
Item 03*	Have you been distracted by thoughts when you should have been concentrating on something else?
Item 04***	Have you had difficulty remembering the names of common things?
Item 05*	Have you had difficulty remembering what date it was?
Item 06***	Have you had difficulty remembering where things are normally kept?
Item 07**	Have you had difficulty remembering whether you had already told someone something?
Item 08***	Have you had difficulty remembering what somebody told you a few minutes earlier?
Item 09***	Have you had difficulty remembering what you were going to say while you were talking?
Item 10**	Have you had difficulty remembering what happened the last few days?
Item 11**	Have you walked into a room but forgotten what you went for?
Item 12***	Have you had difficulty remembering the names of relatives, friends, or other people you see regularly?
Item 13***	Have you had difficulty remembering what you initially were doing if you started to do something else in the meantime?
Item 14**	Have you had difficulty remembering what you were doing when you were interrupted?
Item 15a	Have you had difficulty in concentrating on things, like reading a newspaper or watching television?
Item 16*	Have you been reading something and had to read the same lines again because you were distracted?
Item 17***	Have you had difficulty staying focused on a task or an activity even when it was interesting?
Item 18a	Have you had difficulty remembering things?
Item 19***	Have you had difficulty maintaining concentration even when something really interested you?
Item 20**	Have you had difficulty remembering where you left things, eg, your keys or your wallet?
Item 21**	Have you had difficulty remembering appointments or meetings?
Item 22**	Have you had difficulty remembering what somebody told you a few days earlier?
Item 23**	Have you had difficulty recognizing faces of people you have seen before?
Item 24*	Have you been forgetful?
Item 25*	Have you had difficulty paying attention on a task or a conversation for a longer period of time?
Item 26***	Have you had difficulty recognising relatives, friends, or other people you see regularly?
Item 27***	Have you had difficulty remembering what someone just told you?
Item 28**	Have you had difficulty paying attention as long as you wanted or needed to?
Item 29**	Have you had difficulty paying attention (eg, when watching a movie, reading, or talking to someone)?
Item 30**	Have you had difficulty remembering new information, like a person’s name or simple instructions?
Item 31**	Have you had difficulty remembering to take things you needed with you?
Item 32**	Have you become distracted from a task before finishing it?
Item 33**	Have you had difficulty remembering whether you had already done something?
Item 34***	Have you had difficulty remembering something you had just said?
Item 35*	Have you had difficulty remembering to pass on a message or remind someone of something?
Item 36***	Have you had difficulty maintaining concentration even when doing something important?
Item 37***	Have you had difficulty remembering what you were just thinking?
Item 38**	Have you had difficulty gathering your thoughts?
Item 39**	Have you had difficulty remembering to do the things you had planned to do?
Item 40***	Have you had difficulty remembering what weekday it was?
Item 41**	Have you had difficulty remembering what a text you were reading was about?
Item 42**	Have you had difficulty remembering what you did a few days earlier?
Item 43**	Have you forgotten to do routine things such as turning off the light or locking the door?
Item 44*	Have you had difficulty staying focused on a task or an activity?

^aFrom the EORTC QLQ-C30 Cognitive Functioning (CF) scale. The gray items are related to concentration while the blank items are related to memory.

* Item rated as good CF; ** item rated as moderate CF; *** item rated as poor CF.

Discussion

This study aimed to develop an item bank for the EORTC QLQ-C30 cognitive functioning scale that can be used for CAT. Using a multistep item-selection procedure, a list of 44 items was constructed. All of these items were congruent with the definition and conceptualization of the concept of self-reported cognitive complaints as originally assessed by the QLQ-C30, and compatible with the QLQ-C30 response format. A next step is to evaluate the psychometric characteristics of this concept item list in a large cohort (n > 1000) of cancer patients across Europe (phase IV field-testing). In this phase it will be important to include patients with metastatic and primary brain tumors, for whom these cognitive complaints issues are particularly relevant.

Despite limitations that are inherent to modeling,³³ an advantage of CAT measurement is that the item set can be tailored to the individual patient, thereby increasing measurement precision and reducing response burden, without loss of comparability of results across subjects. A lower response burden is particularly desired for patients with concentration, vigilance, or sustained attention deficits, but also for patients suffering from other symptoms such as fatigue. Moreover, this CAT version will allow direct comparability with reference populations and other studies that used the QLQ-C30 to assess HRQOL. This is accomplished by a strict selection of items that reflect the conceptualization of the original items on cognitive complaints in the QLQ-C30. The majority (79%) of the items address memory, and only 21% address concentration. When using the final item bank in the CAT context, one may consider systematically selecting items to achieve a balance between the 2 subdomains.

We realize that the strict selection of items to fit the QLQ-C30 focus narrows the coverage of cognitive complaints to concentration (including attention) and memory, while cognitive function encompasses more than these domains.^31,34–37 Another limitation is the lack of items on concentration that are rated relevant for patients with poor cognition. This raises the question as to whether it would be useful to develop a separate, more comprehensive cognitive complaints module that could complement the generic QLQ-C30 questionnaire by covering additional cognitive functioning domains. We think that objective testing should be primarily used when the study aim is to determine cognitive functioning. Furthermore, although the EORTC QLQ-C30 is a generic questionnaire developed to measure HRQOL in all cancer patients, the cognitive functioning scale may particularly be relevant for brain tumor patients as well as patients with systemic cancer with a predisposition to metastasize to the brain or those vulnerable to treatment-related cognitive impairments. It is therefore warranted that these patients be included in phase IV of this project.

Similar to the EORTC CAT project is the PROMIS project in the United States.²⁷ The aim of PROMIS is to develop and validate item banks for a range of patient-reported outcome measures relevant for medical research. Currently, PROMIS is developing an instrument to assess self-reported cognitive complaints, including multiple domains.³⁸ Although this instrument has a broad coverage of cognitive domains, it has been developed in the United States and is currently available in English and Spanish only.

In parallel with the development of this item bank for the EORTC QLQ-C30 cognitive functioning scale, item banks for the 13 other QLQ-C30 scales are under development or have recently been completed.^39–41 Completion of all scales will result in a complete CAT version of the original QLQ-C30. A next step is to implement this CAT version in clinical practice and research. To do so, an appropriate software program is required, which is also currently under development by the EORTC QOL Group. This program will include an optimal patient-interface and graphical presentation of the results for both patients and health care providers. Ultimately, this CAT measurement system will facilitate precise and efficient assessment of the HRQOL of cancer patients.

Supplementary Material

Supplementary material is available at Neuro-Oncology Practice online.

Funding

The study was funded by grants from the EORTC Quality of Life Group, and the paper was developed on behalf of the group.

Conflict of interest statement. None of the authors have any conflicts of interest with respect to this work.