Development of the Vision Impairment in Low Luminance Questionnaire

Abstract

Purpose

The purpose of this study was to design and evaluate an instrument for assessing vision-related quality of life appropriate for the specific visual impairment characteristic for all stages of age-related macular degeneration (AMD), with a focus on the low luminance deficit in early/intermediate stages.

Methods

A standardized questionnaire was developed in three steps with participants with early, intermediate, and late AMD: (1) based on in-depth interviews (n = 19) and two focus group discussions (n = 5 each), content was developed followed by 2. (2) The questionnaire development using cognitive debriefing interviews (n = 3) and leading to a preliminary version of the questionnaire. (3) This version was then administered to 127 participants with early, intermediate, and late AMD. Psychometric properties, such as response category functioning (floor and ceiling effects) and targeting of item difficulty to patient ability of the pilot Vision Impairment in Low Luminance (VILL) questionnaire were evaluated using Rasch analysis.

Results

The preliminary VILL questionnaire consisted of 68 items with a 5-step response scale. Several items were removed based on floor/ceiling effects or misfit and a final pool of 37 items remained. The response scale was collapsed to four categories as one category was underutilized. The targeting of the instrument was good with minimal difference in person and item means (0.52 logits). Precision was also good with a person separation index of 3.55 and reliability of 0.93. There was evidence of multidimensionality (eigenvalue of the first contrast = 5.95) in the scale, which could be resolved by splitting the items into subscales including a reading, mobility, and emotional well-being subscale.

Conclusions

Individuals with AMD report difficulties with vision-related activities and functioning under visually challenging conditions at all stages of the disease. These aspects were considered when developing the 37-item VILL, which demonstrates promising psychometric characteristics. Further assessments of reliability and validity are warranted.

Translational Relevance

The VILL questionnaire is a new patient-reported outcome (PRO) measure developed for future use in AMD studies.

Introduction

Age-related macular degeneration (AMD) remains the most common cause of severe visual loss in all high-income countries and we currently lack both interventions to stop or delay onset and progression of early stages of AMD as well as clinical end points to evaluate such interventions in early stages of AMD (i.e. early and intermediate AMD).^1–3 In early and intermediate AMD, patients usually perform well in standard high contrast, high luminance best-corrected visual acuity (BCVA) testing.⁴ However, the most widely used outcome measure in ophthalmic research is BCVA,^5,6 which appears to be largely insensitive to the specific functional impairment in early and intermediate AMD.^7,8 Common visual symptoms in early stages include problems seeing in dim light and at night^9–11 and patients often report difficulties with low contrast and low luminance.^9,12 Previous studies have shown that this vision impairment impacts activities of daily living, falls, and mobility, as well as emotional well-being,^9,13–15 and that self-reported night vision symptoms are associated with low luminance deficit (LLD).¹⁶ The degree of self-reported problems with night vision could be shown to predict both disease progression from early to late AMD as well as a loss of BCVA ≥ 3 lines over a period of 6 years.^17,18

In fact, patient-reported outcomes (PROs) are increasingly used to assess the impact of vision impairment from the patient's perspective, including patient-relevance of changes in retinal structure and functional testing also in a regulatory context.^19–21 Although many questionnaires for assessing vision-related quality of life (VRQOL) and functional ability are available, none have been specifically developed to include vision impairment characteristics of early stages of AMD following available regulatory guidelines.^22,23 Existing measures, such as the Low Luminance Questionnaire (LLQ) or the Night Vision Questionnaire (NVQ), have been developed with patients with AMD but did not follow the US Food and Drug Administration's guidance on PRO development (e.g. including multilevel results of qualitative research to support the instrument's content validity) or have been developed as a generic measure of VRQOL derived from the National Eye Institute Visual Functioning Questionnaire 25 items (NEI-VFQ-25), respectively.^9,17

To enable the development of interventions with the goal to delay or stop onset and progression or reduce visual impairment in early and intermediate AMD, a PRO instrument developed in accordance with existing regulatory guidelines²⁴ is needed to assess the subjective impact and relevance of specific impairment as perceived by patients across all stages of AMD. In order to fill this gap, we designed and evaluated the Vision Impairment in Low Luminance (VILL) questionnaire.

Methods

Participants

All participants were adults (≥ 55 years) and were categorized into early, intermediate, or late AMD based on the Beckman classification system introduced by Ferris et al. based on a clinical assessment including multimodal retinal imaging by a retina specialist.²⁵ Patients were recruited from outpatient clinics. The study was approved by the Institutional Review Board of the University of Bonn (approval ID: 130/16). All patients gave informed consent for study participation. The protocol followed the tenets of the Declaration of Helsinki.

Phases of Instrument Development

The instrument was developed in three steps. In the first phase, content for questionnaire items was identified by reviewing existing instruments, including but not limited to the Functional Reading Independence Index (FRII),²⁶ the Impact of Vision Impairment – Very Low Vision (IVI-VLV) questionnaire,²⁷ the LLQ,⁹ the 10-item NVQ (NVQ-10),¹⁷ and the NEI-VFQ-25.²⁸ Furthermore, 19 in-depth interviews as well as 2 focus groups discussions (FGDs) were conducted with patients with early, intermediate, or late AMD. Interviews and FGDs were conducted by a trained interviewer using an interview guideline. In-depth interviews were conducted either in person or by telephone, depending on participant preference. For FGDs, participants with similar levels of disease severity were grouped to foster social facilitation, a common approach in FGDs.²⁹ Both in-depth interviews and FGDs were audiotaped and transcribed verbatim. Transcripts were examined using an inductive analytical approach. This is an iterative process, which involves broadly coding data into themes and subsequently continually revising these themes as further transcripts are analyzed until thematic saturation occurs. Data were analyzed qualitatively using NVivo (version 11; QSR International, Burlington, MA, USA) for structuring and visualization purposes.³⁰ In the second phase, this content was used to develop a preliminary VILL questionnaire with 75 items and a five-step response scale ranging from “very” to “not at all” (items 1–32) and “always” to “never” (items 33–68). One additional response option captured whether items were applicable to participants (i.e.: “not applicable”). This was followed by cognitive debriefing interviews to ascertain unambiguous phrasing of items and response scales as well as appropriateness of content based on a standardized guideline and during which patients were encouraged to think aloud.³¹ In a third phase, the resulting 68 items pilot VILL was administered to 127 patients with early, intermediate, and late AMD. Best-corrected visual acuity (BCVA) of these patients was assessed according to the early treatment diabetic retinopathy study (ETDRS) method.³²

Psychometric properties of the pilot VILL questionnaire, such as response distribution per item (floor and ceiling effects) and targeting of item difficulty to participant ability, were determined using Rasch analysis. Supplementary Table S1 provides an overview of all items tested and the final items retained for the VILL.

Psychometric Evaluation of the Pilot VILL

Rasch analysis is a psychometric method that mathematically describes the interaction between respondents and test items and applies a model that the pattern of participants’ responses should satisfy.^27,33,34 It transforms ordinal scales into interval-level scales (expressed in logits). This allows to calculate item difficulty (item measure) in relation to person ability (person measure) by placing both in the same linear continuum.^35,36 To assess the psychometric properties of the pilot VILL, we used the following criteria.

Threshold Ordering: We assessed the response category threshold ordering to determine whether the categories used to rate VILL items are valid. Over- or underutilization of response categories and the ability of participants to discriminate between the response categories were assessed. Disordered thresholds, if evident, were addressed by collapsing categories.^37,38

Precision of the Instrument: The ability of the scale to discriminate between different levels strata of person ability was assessed using person separation index (PSI) and person reliability (PR) scores. Values of > 2.0 and > 0.8, respectively, were considered adequate and represented the capacity of the scale to distinguish three levels of person ability.^39,40

Unidimensionality: Unidimensionality describes the ability of a scale to measure a single underlying trait and whether the items’ “fit” the underlying trait which was assessed twofold. First, we determined how well each item “fits” or “misfits” the underlying trait through an “infit” mean square standardized residuals (MNSQ) statistic.⁴¹ An infit MNSQ value of 1 is ideal and up to 1.3 is acceptable. High fit values are regarded as misfitting (noisy and erratic) and values below 0.7 as overfitting (muted).³⁹ Second, we conducted a principal component analysis (PCA) of the residuals in order to test for local independence. The PCA of residuals for the first factor should explain at least 50% of the variance and the first contrast of residuals should be < 2.5 eigenvalue.^42,43

Targeting: The targeting of the instrument (i.e. how well item difficulty corresponds to the person's ability), was determined by inspecting the person-item map and calculating the difference between person and item mean logits. A difference of > 1.0 logits indicates that the difficulty of the respective item does not adequately target the ability of the sample.^38,44

Differential Item Functioning (DIF): Each item was assessed for DIF, which is a statistical method for detecting whether sample subgroups (e.g. gender and age groups) respond systematically different to certain items, despite having a similar underlying ability. A DIF contrast of > 1.0 logits is notable and suggests that the item may be biased for some participant subgroups. We assessed DIF for gender and age groups < 75 years and ≥ 75 years (based on the median age of the sample). Only significant DIF values (P < 0.05) were reported.

Rasch analysis was performed using commercial software (version 3.92.1.2; Winsteps Software, Chicago, IL).⁴² The Andrich rating scale model was used for analysis.⁴³ Two rating scales were applied to this questionnaire because there were two sets of response options with different characteristics.

Statistical Analysis

Commercial statistical software (SPSS Version 25; SPSS Science, Chicago, IL) was used to analyze the data.⁴⁵ Descriptive statistical analyses were performed to characterize the participants’ sociodemographic and clinical characteristics. An unpaired t-test was used to compare means of the VILL scores among age groups, sex, two different levels of visual acuity (VA), AMD stage, and the self-reported presence of depression (“Are you known to have depression?”), to support discriminant validity of the instrument. A P value < 0.05 was considered statistically significant.

Results

Focus Groups Discussions, in-Depth Interviews, and Cognitive Debriefs

Nineteen patients with early, intermediate, or late AMD participated in in-depth interviews. Five patients with early AMD and five patients with late AMD participated in one FGD each. Table 1 shows that both subgroups had similar demographic characteristics.

Table 1.

Demographic Characteristics of In-Depth Interviews and Focus Group Discussions

	AMD Stage
	Early	Intermediate	Late
In-depth interviews
N	3	10	6
Age, mean (SD)	76.3 (8.5)	72.4 (9.3)	79.6 (8.5)
Gender, % (n)
Female	66.6 (2)	70 (7)	83.3 (5)
Male	33.3 (1)	30 (3)	16.6 (1)
FGD
N	5	–	5
Age, mean (SD)	76.8 (2.6)	–	78.6 (6.9)
Gender, % (n)		–
Female	80 (4)	–	100 (5)
Male	20 (1)	–	0 (0)

FGDs and in-depth interviews revealed that emerging themes were related to difficulties with reading, accessing information, and recognizing people in everyday situations under low contrast and/or low lighting levels (e.g. newspaper, price tags in shops, advertisements with colorful backgrounds, and encounters outside in the dark). Activities related to mobility at dusk or at night including driving as well as safety while engaging in mobility were also mentioned frequently. Moreover, in both the interviews and the FGDs, common themes were socio-emotional distress due to the concern of losing independence, worsening vision in the future, and the resulting impact on everyday life. Based on the results of the qualitative analysis we developed a draft questionnaire consisting of 75 items with a 5-step response scale, including the domains of “reading and accessing information,” “orientation and mobility,” “safety,” and “socio-emotional well-being.” After conducting cognitive debriefing interviews with 3 patients with AMD to assess comprehensibility and appropriateness of each question, 7 items were removed because patients felt they were difficult to understand and not relevant to their daily lives, resulting in a revised version of the questionnaire with 68 items with a 5-step response scale.

Psychometric Evaluation of the VILL Questionnaire

The pilot VILL was administered to 127 patients with early, intermediate, and late AMD and the validity, reliability, and dimensionality of the questionnaire were assessed using Rasch analysis (Table 2). One patient was excluded from the analysis because the interview was not completed. In accordance with the two response categories, two rating scales were applied (items 1–32 referring to “difficultly,” ranging from “very” to “not at all,” and items 33–68 referring to “bother,” ranging from “always” to “never”). The 68-item version of the VILL had disordered thresholds in both rating scales, suggesting that the use of the 5 initial response options was suboptimal. For both rating scales, category 1 was unlikely to be chosen. Consequently, categories 0 and 1 (“very”/“always” and “considerable”/“frequent”) were merged, which resulted in 4 final response options for each scale. The VILL displayed good discriminant ability with PSI and PR values of 4.2 and 0.95, respectively. The targeting of the VILL was slightly suboptimal with a difference in person and item means of 0.52 logits. The PCA yielded evidence of multidimensionality, because the first factor explained < 50% (47.3%) of the variance and had an eigenvalue of 5.95. This suggests the existence of a second dimension. Moreover, six items (items 39, 40, 50, 52, 64, and 86) demonstrated substantial misfit (MNSQ > 1.3). Twenty-nine items revealed floor (27 items) or ceiling effects (2 items) and for two items a large proportion of participants (> 30%) indicated that these items were not applicable. In total, 31 items were removed due to the reasons stated above and 37 items remained. For the 37-item version of the VILL, PSI and PR were 3.55 and 0.93, respectively, implying that three levels of person strata can be detected. Targeting was good, with a difference between person and item means of 0.29. The person-item map indicated a good item coverage for the majority of the sample (Figure). Six items (items 3, 8, 16, 18, 44, and 49) demonstrated misfit with MNSQ values < 0.7, however, removal of these items did not improve fit statistics. There was still evidence of multidimensionality in the PCA with the first factor explaining 44.7% of the variance and an eigenvalue of 4.5 for the first contrast, indicating the presence of at least 2 subscales. Four items (items 58, 59, 61, and 66) loaded positively (correlation > 0.4) onto the first contrast. These items referred to aspects of emotional well-being, suggesting that they belong to the same domain. Therefore, we split the items into three subsets. An emotional well-being subscale with the above-mentioned four items, a reading and accessing information subscale (20 items), and a mobility and safety subscale (13 items). More details on the items and the assignment into the subscales can be found in the Supplementary Material. The reading and mobility scales showed good PSI and PR values (2.68 and 0.88 for the reading scale and 2.03 and 0.80 for the mobility scale). However, the emotional well-being scale returned unsatisfying results regarding PSI and PR with values of 1.13 and 0.55, respectively. No item displayed misfit in the reading and mobility scale. The targeting of both scales was slightly suboptimal with a difference in person and item means of 0.88 and 0.80 logits, respectively, but within an acceptable range. The reading scale showed minimal evidence of multidimensionality with PCA for the first factor explaining > 50% (53.9%) and an eigenvalue for the first contrast of 2.6. For the mobility scale, PCA of the residuals was 54.9%, and the first contrast of the residuals was 2.7 eigenvalue, which is acceptable for the requirements of unidimensionality. No significant DIF was found for gender or age in either of the subscales.

Table 2.

Fit Parameters of the VILL Questionnaire With 68 Items (VILL-68), the VILL Questionnaire Without Misfitting Items (VILL-37), the Reading Scale (20 items), the Mobility Scale (13 items) and the Emotional Scale (4 items) Compared With Rasch Model Requirements

Parameters	Rasch Model Requirements	VILL-68	VILL-37	VILL –Reading and Accessing Information	VILL – Mobility and Safety	VILL –Emotional Well-Being
Misfitting items, n	0	Items 40, 50, 51, 52, 39, 64, 68	Items 49, 18, 8, 16, 3, 44	None	None	All items < 0.7
PSI	>2.0	4.2	3.55	2.68	2.03	1.13
PR	>0.8	0.95	0.93	0.88	0.80	0.55
Difference in person and item mean	<1	0.52	0.29	0.85	0.75	0.74
Variance by the first factor	>50%	47.3%	44.7%	53.9%	54.9%	68.4%
PCA (eigenvalue for 1st contrast)	<2.5	5.95	4.5	2.6	2.5	1.5
Differential item functioning (Item number [DIF contrast])	< 1.0, P < 0.05
Gender		na	None	None	None	None
Age group (< 75 to ≥ 75)		na	None	None	None	None

na = not assessed.

Figure.

Person-ITEM map for the VILL questionnaire with 37 items.

Association of the VILL Questionnaires Scores with Sample Characteristics

Rasch analysis was used to generate person measures in logits for all participants with higher scores indicating poorer VRQOL. The overall mean score was −0.28 (SD ± 0.84) logits. Mean person measures for the 3 subscale scores for different groups are shown in Table 3. The mean person measures of the overall score and two subscales (reading and accessing information, and emotional well-being) were significantly lower in participants with early/intermediate than in those with late AMD (P ≤ 0.025; see Table 3). The orientation and mobility subscale did not differ between these subgroups. There was no difference in the overall VILL scores by age groups, sex, level of visual impairment, or self-reported depression (all P values > 0.05; see Table 3). Likewise, there was no significant difference in any of the sample characteristics regarding the subscale scores for the reading and mobility subscale, although the difference between age groups in the latter one almost reached significance (P = 0.055). For the score of the emotional scale, there was a significant difference by age groups and sex, but not for visual impairment or depression.

Table 3.

VILL Questionnaire Scores by Sample Characteristics, n = 126

		VILL-37		VILL – Reading and Accessing Information		VILL – Mobility and Safety		VILL – Emotional Well-Being
Parameters	n	Mean ± SD	P Value*	Mean ± SD	P Value*	Mean ± SD	P Value*	Mean ± SD	P Value*
Age
<75	44	−0.31 ± 0.78	0.824	−0.60 ± 0.92	0.698	−0.65 ± 1.27	0.326	−0.33 ± 1.42	0.276
75 +	81	−0.28 ± 0.88		−0.53 ± 0.87		−0.45 ± 1.02		−0.60 ± 1.21
Gender
Female	88	−0.26 ± 0.84	0.639	−0.50 ± 0.84	0.633	−0.43 ± 1.1	0.184	−0.33 ± 1.24	0.034
Male	38	−0.34 ± 0.85		−0.58 ± 0.89		−0.72 ± 1.01		−0.86 ± 1.33
AMD stage
Early/intermediate	81	−0.42 ± 0.89	0.020	−0.71 ± 0.90	0.009	−0.59 ± 1.27	0.301	−0.68 ± 1.21	0.025
Late	45	−0.05 ± 0.69		−0.28 ± 0.78		−0.38 ± 0.73		−0.15 ± 1.36
Visual acuity
≥0.5 logMAR	6	−1.05 ± 1.07	0.577	−0.45 ± 1.04	0.739	−0.38 ± 0.85	0.743	0.08 ± 1.54	0.268
<0.5 logMAR	117	−0.30 ± 0.84		−0.57 ± 0.88		−0.53 ± 1.11		−0.52 ± 1.27
Self-reported depression
Yes	13	0.05 ± 0.51	0.121	−0.18 ± 0.69	0.109	−0.18 ± 0.45	0.254	−0.92 ± 1.45	0.205
No	109	−0.33 ± 0.87		−0.60 ± 0.91		−0.56 ± 1.17		−0.57 ± 1.27

^*Unpaired t-test.

Discussion

Patients with AMD report difficulty with vision-related activities and functioning under visually challenging conditions at all stages of the disease. These include reading, social interaction/recognizing people, mobility/safety, and the socio-emotional impact of these difficulties. These aspects were considered when developing a novel PRO instrument to capture patient-reported difficulty with vision-related activities and functioning under visually challenging conditions. Using a large item pool generated with participant and expert input as well as Rasch analysis the resulting VILL questionnaire is able to discriminate among three different strata of ability and the measurement was not affected by sample characteristics, such as age, sex, or depression in our sample but captured differences between AMD disease stages.

Many of the proposed criteria for quality of health status questionnaires, such as content validity, criterion validity, internal consistency, no floor or ceiling effects, and good interpretability, are met with the VILL-37.⁴⁶ However, the 37-item version still displays multidimensionality, indicating that splitting the scale into subscales, including a reading, a mobility, and an emotional subscale, is reasonable and necessary for further psychometric evaluation in a larger sample. Persisting issues with the emotional scale could be solved by removing all four items belonging to this scale. However, this results in a lack of any information on the socio-emotional impact of AMD and would render the questionnaire a measure of functional impairment and visual difficulty only. Although there is no consensus on a definition of VRQOL, there is considerable agreement among experts that it should encompass psychological or psycho-social well-being.^47,48 Therefore, the emotional subscale of the VILL was retained and requires further evaluation. None of the items in the VILL-37 exhibited significant DIF by gender or age group (< 75 / ≥ 75 years) in our sample.

Patients with AMD, particularly those with early or intermediate stages, commonly report visual difficulties in dim light and under low contrast.^49–52 These common visual problems can be verified psychophysically.^9,49–51 Rod photoreceptors are selectively vulnerable to dysfunction and degeneration in the early stages of AMD and studies have demonstrated that patients in the early stages have, for example, impaired rod-mediated dark adaptation.^53–55 Focusing on this, the VILL questionnaire will be useful as a patient-centered tool for assessing VRQOL and functional impairment in patients with AMD, in particular in early and intermediate disease stages as opposed to late AMD. Person measures of the reading and emotional subscales were sensitive to disease severity in our sample. This supports the responsiveness of the VILL but it is unclear from the available data why the mobility subscale did not differ between early stages of AMD and late AMD. Patients with AMD might affect reading and near work to a larger extent than mobility but additional studies are needed to explore this in more detail. Unlike disease stage, photopic VA was not significantly associated with person measures of the VILL subscales. This corresponds with the design of the VILL items, which focus on the characteristic functional deficit under low contrast and low luminance in AMD.

Strengths of our study include the use of qualitative research, a literature review, and expert input to create a large item pool together with patients with various stages of AMD. Therefore, items have real-world validity and are patient relevant. Item selection was informed by cognitive debriefing interviews and further pilot data. Patients were clinically assessed and uniformly staged according to current clinical reference standards. Another strength is the use of Rasch analysis and the final instrument could be shown to satisfy requirements of the Rasch model. Rasch analysis provided several useful indicators of scale category organization, such as the optimal number of response options and the validity and functioning of the rating scale.^27,56,57 As a result of disordered thresholds, we collapsed the two used rating scales from five to four response options. This is in line with previous findings that ophthalmic questionnaires function optimally with no more than five and often four response categories.⁵⁸ There are limitations of our study, which include the limited sample size and persisting psychometric issues, such as the poor functioning of the emotional well-being subscale as well as the age structure of the sample with a higher proportion of late AMD in older participants. At this stage in the questionnaire development, the scale was retained as this ensures that the PRO extends insights from conventional psychophysical assessments of visual functioning to the affective aspect of VRQOL in patients with AMD. Only few participants in our study had early AMD; as a consequence, the content of the items is not specific to early AMD and no conclusions regarding the reliability and validity of the VILL in a sample with only early AMD can be drawn from our data. Because the distinction of early versus intermediate AMD is solely made by structural criteria, not functional criteria, we collapsed both groups – early and intermediate AMD – into an “early stages of AMD” group. We did not assess the VILL's test-retest reliability and the association with functional measures of vision. For this as well as an evaluation of clinical utility, further evaluation of the VILL in a larger sample size is required. The VILL-37 itself is currently limited in use by its length and associated participant burden. However, the goal of this study was to design and assess psychometric characteristics of the instrument and our results support the validity of the tool for use in AMD. Additional studies will focus on further item reduction.

In conclusion, patients with AMD report difficulty with vision-related activities and functioning under visually challenging conditions at all stages of the disease. These aspects were considered when developing the 37-item VILL, which has demonstrated good psychometric characteristics. Further assessments of reliability and validity in different studies are ongoing.