Data quality, floor and ceiling effects, and test–retest reliability of the Mild Cognitive Impairment Questionnaire

Abstract

Background

The Mild Cognitive Impairment Questionnaire (MCQ) is a 13-item measure that assesses health-related quality of life (HRQoL) in people with mild cognitive impairment (PWMCI); it has two domains assessing the emotional and practical effects.

Objective

The aim of this study was to assess the psychometric properties of the MCQ.

Design

This is a longitudinal questionnaire-based study.

Setting

The participants were recruited from the memory clinics and research databases in the South of England.

Subjects

A total of 299 people aged 50 years and older with a diagnosis of mild cognitive impairment confirmed within the preceding 12 months.

Methods

MCQs were distributed to patients in memory clinics and those listed on research databases. Participants who returned completed questionnaires were sent a second copy of the MCQ to return 2 weeks after receiving the first questionnaire.

Results

Five hundred and seven questionnaires were distributed; response rates were 68.2% initially and 89.2% for the second questionnaire. From the returned questionnaires, response rates for each item were high (>98%) and a full range of responses for each item was received with no evidence of significant floor or ceiling effects. Internal consistency reliability for both scale scores at both time points was good, with Cronbach’s a≥0.84 in all cases. Test–retest reliability was excellent for both domains with the intraclass correlation coefficients of 0.90 and 0.92 for the practical and emotional domains, respectively. Paired sample t-tests also confirmed the stability of scale score distributions over time.

Conclusion

The MCQ has robust psychometric properties, which make it suitable for assessing HRQoL in PWMCI, including comparison of group level data in intervention studies.

Introduction

Mild cognitive impairment (MCI) is a common condition, with a prevalence of ~3% in the general older population,1 and the rates of diagnosis are increasing as a result of recent government policies encouraging the early diagnosis of dementia.2,3 MCI has been shown to be associated with significant emotional and practical challenges for those living with MCI.4–6 Despite this, until recently, no validated patient-reported outcome measure (PROM) for the assessment of health-related quality of life (HRQoL) existed for MCI. This is important for the following two reasons: first, assessment of HRQoL is an important part of the clinical assessment of patients in everyday practice, particularly given the evidence that the condition has an adverse effect on quality of life and second, there is growing interest in conducting trials of potentially disease-modifying treatments for dementia in patients with MCI who have a high rate of conversion to dementia, as a study population with a “pre-dementia” condition. It has been noted in the literature that, up to now, no PROMs developed specifically existed for MCI7,8 and that the lack of consistency in outcome measures used has made it difficult to compare and interpret the results of many of the interventional studies that have been carried out in this field.8 PROMs, defined by the UK Department of Health as “measures of a patient’s health status or health-related quality of life […] typically short, self-completed questionnaires”9 are the ideal tool for use in this type of study, particularly as many of the assessments of cognition used currently are not sufficiently sensitive to measure the mild degree of cognitive impairment seen in MCI. In addition, both the European Medicines Agency10 and the US Food and Drug Administration11 have issued guidance for the pharmaceutical industry regarding the use of PROMs in medical product development.

In order to address this deficiency, the Mild Cognitive Impairment Questionnaire (MCQ) was developed in a previous study.12 The development process involved semistructured interviews with people with MCI (PWMCI) and their carers. The data from these interviews were analyzed using qualitative methodology, and a draft version of the MCQ was produced. This version was refined following discussion with focus groups (consisting of PWMCI and their carers), and the final version was administered to a large number of PWMCI.19 The results obtained were analyzed using factor analysis, which resulted in a 13-item measure including two scales measuring “emotional effects” and “practical concerns”. Analysis of the data in that study showed that the MCQ has good psychometric properties in terms of internal consistency reliability (as measured by Cronbach’s a) and validity.

The aim of this study was to further assess the psychometric properties of the MCQ, including test–retest reliability, to facilitate its use both in clinical practice and, as a potential outcome measure, in intervention studies.

Methods

The MCQ, together with some basic demographic questions, was distributed (in person) to people diagnosed with MCI in memory clinics and (by post) to people on research databases in Berkshire, Essex, Hertfordshire, and Oxfordshire (UK). This was designated as “Time Point 1”. People invited to take part in the study were aged 50 years or older with a diagnosis of MCI confirmed in a memory service (using whichever criteria the diagnosing clinician had applied) within the 12 months preceding recruitment. The memory clinics and research databases in Oxford and Essex from which the majority of participants were recruited, both for the study in which the MCQ was originally developed,12 and for this study, all used the well-known diagnostic criteria for MCI set out by Petersen et al.13 Participants were asked to complete and return the questionnaire within 2 weeks, and those recruited from research databases received one written reminder about the study if their questionnaire had not been received after 2 weeks. Time Point 1 recruitment was carried out over a 24-month period between May 2014 and May 2016.

The participants who returned questionnaires were sent a second copy of the MCQ to complete and return 2 weeks after receiving the first questionnaire (ie, Time Point 2).

MCQ data were analyzed using the following criteria:

1. Data completeness, ie, rates of item-level missing data;

2. Response distributions for each item including floor and ceiling effects;

3. Features of scale score distributions at each time point;

4. Internal consistency reliability of scale scores;

5. Test–retest reliability of the scale scores between the two time points;

6. Comparison of the scale score distributions between the two time points.

Consent and approvals

Approval of this study was granted by the North of Scotland Research Ethics Service, REC Reference 14/NS/0031.

All participants gave (or had already given) written informed consent to their details being stored and used for the purposes of research. Consent to participate in the study was implied by completion and return of the study questionnaires – this was clearly explained in the study information supplied to the participants.

Results

Response rates and demographics

Five hundred seven questionnaires were distributed at Time Point 1; of these, 346 (68.2%) questionnaires were returned. Three hundred forty-five questionnaires were distributed at Time Point 2; of these, 308 (89.2%) questionnaires were returned.

Forty-seven completed questionnaires were excluded as the participants did not meet the inclusion criteria (or there was insufficient information included in the demographic information to ensure that they did); therefore, 299 completed questionnaires were included in the analysis.

The characteristics of the participants included in the study are listed in Table 1.

Table 1.

Characteristics of study participants

Characteristics	Number/value	Percentage
Gender
Male	162	54.2
Female	137	45.8
Age (years)
Range	50–95	–
Median	76.0
Interquartile range	70.0–82.0
Mean	75.3	–
Standard deviation	8.8	–
Time from diagnosis confirmation (months)
<1	98	32.8
1–3	104	34.8
4–6	46	15.4
7–12	51	17.1
Ethnicity
Asian	7	2.3
Black	4	1.3
White	287	96
Mixed	1	0.3

Data completeness and response distributions

Seventeen (5.7%) respondents did not complete all 13 items of the MCQ at Time Point 1, and 15 (4.9%) respondents did not complete all items at Time Point 2; therefore, it was not possible to calculate both dimension scores for these participants. However, as listed in Table 2, response rates to each individual question were high.

Table 2.

Item completeness and response distribution for each MCQ item (Time Point 1)

Item	n	% missing	Response category (%)
			Never	Rarely	Sometimes	Often	Always
P1	296	1.0	2.0	9.0	45.5	33.8	8.7
P2	298	0.3	28.1	27.8	26.8	14.0	2.7
P3	296	1.0	10.0	23.7	42.8	19.1	3.3
P4	296	1.0	12.0	26.4	32.4	19.4	8.7
P5	296	1	12.4	19.7	34.1	25.4	7.4
P6	298	0.3	26.1	22.7	30.1	14.4	6.4
P7	297	0.7	11.4	20.4	36.1	23.4	8.0
E8	295	1.3	8.0	17.1	29.1	34.1	10.4
E9	296	1.0	7.7	17.7	32.1	31.1	10.3
E10	295	1.3	18.7	22.7	31.8	18.7	6.4
E11	295	1.3	25.1	21.7	34.1	13.0	4.7
E12	296	1.0	27.4	24.1	27.8	15.1	4.3
E13	295	1.3	6.7	9.7	33.4	28.1	20.7

Abbreviation: MCQ, Mild Cognitive Impairment Questionnaire.

The distribution of responses for each MCQ item is also given in Table 2. Response distributions, as might be expected for a “mild” condition, tended to be skewed toward better health, but a full range of responses to all items was observed and none had floor or ceiling effects >29%.

Scale score features for the two dimensions of the MCQ

Results from the administration of the MCQ at the two time points are listed in Table 3.

Table 3.

Descriptive statistics, score distributions, and internal consistency reliability for the MCQ domains at each time point

Descriptor	Time Point 1		Time Point 2
	Practical	Emotional	Practical	Emotional
n	289	289	251	257
Mean	37.0	38.7	36.3	38.3
Median	37.1	40.0	37.1	40.0
Minimum	2.9	0	0	0
Maximum	80	80	77.1	80
25th percentile	27.5	23.3	22.9	23.3
75th percentile	45.7	53.3	48.6	53.3
Cronbach’s α	0.84	0.90	0.87	0.90

Abbreviation: MCQ, Mild Cognitive Impairment Questionnaire.

Cronbach’s a coefficients14 were calculated to estimate the internal consistency reliability of each scale score, as listed in Table 3; a values >0.7 are recommended for group-level hypothesis testing,15 and a values >0.9 may mean that the measure may be appropriate for use at an individual level.16

Test–retest reliability

Intraclass correlation coefficients (type [3.1])17 were calculated for each of the scale scores between the two time points in order to evaluate test–retest reliability. For the practical scale, intraclass correlation coefficient is 0.90 (95% CI 0.87–0.92, P<0.001), and for the emotional scale, intraclass correlation coefficient is 0.89 (95% CI 0.86–0.92, P<0.001). This indicates excellent test-retest reliability for both domains.

Paired sample t-tests were performed to evaluate whether there was any change in the distribution of the scale scores between the two time points. These revealed no significant differences (practical: n=243, t=0.61, and P=0.54; emotional: n=249, t=0.025, and P=0.98).

Discussion and conclusion

Several measures were used for further assessment of the psychometric properties of the MCQ. First, data completeness was assessed; this was good for all 13 items indicating that there were no items to which a high proportion of subjects did not respond. This is important as high levels of nonresponse can indicate problems with an item such as subjects finding it difficult to understand or upsetting.

Second, response distributions for each item were analyzed and this showed that all response categories were used for all items with no significant floor or ceiling effects, indicating that the items tap a wide range of HRQoL effects. This is an important feature as it is known that MCI is a heterogeneous condition, and consequently, its effects can be very variable between individuals. Generally, a total of <40% respondents selecting “never” or “always” indicated that an item does not show significant “floor” or “ceiling” effects, respectively and it was hence reassuring that the vast majority of items in the MCQ had floor effects <30% and ceiling effects <15%. Third, features of scale score distributions, such as the spread of scores generated, were assessed at each time point, which again indicated that a wide range of effects are tapped by the scores.

Fourth, the internal consistency reliability of the scale scores was assessed using Cronbach’s a. High levels of internal consistency reliability provide greater confidence when using a measure to compare treatment groups, for example, in randomized intervention trials such as those using populations of PWMCI as study subjects in trials of potentially disease-modifying dementia treatments. As discussed in the “Introduction” section, such trials are becoming increasingly popular and there is currently a dearth of instruments appropriate for use as outcome measures. In all cases, the Cronbach’s a exceeded 0.7 indicating the measure’s usefulness in group-level hypothesis testing. This means that the measure could be used as a reliable tool to assess the effect of interventions in studies using PWMCI as subjects. For the emotional domain, the Cronbach’s a for the datasets at both time points was 0.9, which might support the use of this scale for assessment at the individual patient level in clinical practice, another important potential use of the MCQ. Ongoing research regarding sensitivity to change of the MCQ will help to establish whether it would be appropriate to use the MCQ for individual clinical analysis.

Fifth, test–retest reliability of the scale scores between the two time points was examined using intraclass correlation coefficients; these showed excellent reliability for both domains indicating that the two scales produce stable scores over time (assuming no clinical change). It is important that a measure used to assess change over time produces stable scores in the absence of clinical change so that any change in score that occurs can be confidently attributed to a change in clinical condition rather than measurement error. In this case, the high reliability of both domains within a time frame (2 weeks) where no significant clinical change would be expected in MCI suggests that the MCQ produces stable scores over time in the absence of clinical change. Finally, scale score distributions between the two time points were compared using t-tests, which also confirmed that scale score distributions remained stable over time.

It should also be noted that the response rates to the questionnaire were relatively high (~70% at the first time point and ~90% at the second), which suggests that the practicalities of completing the MCQ are not unduly challenging for PWMCI. These response rates are relatively good when compared with similar work: response rates for such surveys rarely exceed 70% and are often considerably lower.

A possible limitation of the study is the fact that there was potential for variability in the definition of MCI used when diagnosing participants. The decision to allow participants to be recruited having been diagnosed with MCI using “whichever criteria the diagnosing clinician had applied” was intentionally made so that participants recruited to the study would reflect “real world” clinic populations and ensure that results were applicable to them. Despite the fact that the inclusion criteria for the study allowed variable definitions to be used for diagnosis, the majority of participants (both in this study and the one in which the MCQ was originally developed12) were recruited from memory clinics and research databases, which applied the Petersen et al13 diagnostic criteria. As a result, the participants were most likely relatively homogeneous with respect to MCI diagnosis. In addition, although there was a wide range of age among the participants (50–95 years), the interquartile range was much narrower (70–82 years), which also suggests reasonable homogeneity among the subjects and correlates roughly with the age at which MCI diagnosed by the Petersen et al18 criteria has a peak prevalence.

The MCQ has been developed using robust qualitative and quantitative methodology as described previously.12 This article provides further evidence that the MCQ has good psychometric properties that make it suitable for use in assessing areas of HRQoL relevant to people living with MCI and comparing treatment groups in intervention trials. Further work, which is currently ongoing, will provide more evidence regarding sensitivity to change and, potentially, use of the measure at the individual patient level.