FLORA™: Phase I development of a functional vision assessment for prosthetic vision users

Abstract

Background

Research groups and funding agencies need a functional assessment suitable for an ultra-low vision population in order to evaluate the impact of new vision restoration treatments. The purpose of this study was to develop a pilot assessment to capture the functional vision ability and well-being of subjects whose vision has been partially restored with the Argus II Retinal Prosthesis System.

Methods

The Functional Low-Vision Observer Rated Assessment (FLORA) pilot assessment involved a self-report section, a list of functional vision tasks for observation of performance, and a case narrative summary. Results were analyzed to determine whether the interview questions and functional vision tasks were appropriate for this ultra-low vision population and whether the ratings suffered from floor or ceiling effects. Thirty subjects with severe to profound retinitis pigmentosa (bare light perception or worse in both eyes) were enrolled in a clinical trial and implanted with the Argus II System. From this population, twenty-six subjects were assessed with the FLORA. Seven different evaluators administered the assessment.

Results

All 14 interview questions were asked. All 35 functional vision tasks were selected for evaluation at least once, with an average of 20 subjects being evaluated for each test item. All four rating options -- impossible (33%), difficult (23%), moderate (24%) and easy (19%) -- were used by the evaluators. Evaluators also judged the amount of vision they observed the subjects using to complete the various tasks, with vision only occurring 75% on average with the System ON, and 29% with the System OFF.

Conclusion

The first version of the FLORA was found to contain useful elements for evaluation and to avoid floor and ceiling effects. The next phase of development will be to refine the assessment and to establish reliability and validity to increase its value as a functional vision and well-being assessment tool.

A variety of vision restoration approaches for the treatment of blindness are being developed, including (but not limited to) stem cell therapy, gene therapy, and visual prosthetics [1-4]. As more therapies are tested in clinical trials and achieve commercial approval, the need for standardized assessments to evaluate the effectiveness of vision restoration will become pressing[5].

The subject population receiving a visual prosthesis can be described as having ultra-low vision, which is defined as: “visual impairment that impacts most daily living activities involving any visual shape recognition; in terms of visual acuity (VA), it corresponds to the inability to discern the largest Early Treatment of Diabetic Retinopathy Study letters at 0.5 m, i.e., VA - 20/1600 or 6/480”[6]. Ultra-low vision, whether native or restored, cannot be objectively evaluated by the functional vision assessments that are commonly used by low-vision therapists. The assessments of functional vision that are available (for example, the NEI-VFQ-25[5] or the Massof Activity Inventory[7]) have only a few items that can be completed by those with ultra-low vision (locate a source of light), with the majority of test items requiring higher levels of spatial vision (read, recognize faces, identify colors)[8-12]. An assessment that can capture the functional visual abilities of subjects in clinical trials with these novel treatments needs to be developed[13]. This is the ultimate goal of the work reported here.

While a general need for assessment of functional ultra-low vision has been recognized, the development of this assessment was specifically prompted by the needs of one company that manufactures a retinal prosthesis. Second Sight Medical Products (Sylmar, CA) has sponsored a clinical trial and received market approval of their retinal prosthesis (Argus® II Retinal Prosthesis System) in the European Economic Area and the United States. This was the first vision-restoration device or treatment on the market for patients blinded by Retinitis Pigmentosa. It creates visual percepts by electrically stimulating the remaining cells of the retina (after the photoreceptors have been destroyed by the disease). In the clinical trial, subjects’ visual function was evaluated in computer-based tests, and their functional vision was evaluated in lab-based simulations of orientation and mobility tasks (e.g., following a line on the floor)[2]. During discussions with the US Food and Drug Administration (FDA), it became clear that an additional assessment of real-world functional vision and well-being was necessary to complete the picture of the effect of vision restoration on subjects’ lives.

The goal of our work was to begin the development of an assessment that would capture real-world function at the level of vision provided by the Argus II System and other new treatments. Identifying possible items for the assessment, discussing these items with subjects, and seeking the observations of qualified rehabilitation personnel were the initial steps of the process and are the purpose of this report. Our ultimate goal is to develop a standardized tool that can be used in clinical trials of other devices and blindness treatments, to demonstrate the impact of the treatment, and to identify areas where rehabilitation might increase the functional skills of patients with ultra-low vision.

The assessment that was developed for the Argus II clinical trial is known as the Functional Low-Vision Observer Rated Assessment (FLORA) and is available for download at http://www.2-sight.eu/ee/landing-flora.

The purpose of this article is to describe the first steps in the development of the FLORA and to evaluate how the interview questions, functional vision tasks, and rating scale were used when the FLORA was administered to clinical trial subjects. The intent was to determine if the assessment contained items that would be useful for assessing the type and amount of ultra-low vision that is provided by the Argus II system.

Methods

Development of the FLORA

The first phase in the development of this new assessment was to assemble a team of experts in blind and low-vision rehabilitation and the Argus II System (the authors) to study the problem of assessing ultra-low vision and to draft an assessment. During a two-day meeting, the group discussed the draft assessment and suggested revisions. Over the next several weeks, the team continued to revise the FLORA. During the development, we adhered to a few underlying principles, including the importance of self-report (to gain a greater understanding of the subjects’ own perceptions of how the device was affecting them, and to capture the unique ways to use the device that they discovered on their own); the need for expert-observer-rated tasks to confirm or contest the subjects’ self-report; and the desire to capture important well-being benefits (or detriments) that are difficult to measure but are as important as functional outcomes. Several challenges in assessing this population were apparent as we designed the assessment. Addressing each challenge shaped the development of the FLORA.

First, all of the Argus II clinical trial subjects had adapted to life without sight; some had been blind for decades. Therefore, they all had established coping methods, ranging from complete independence with mobility aids to dependence on a sighted guide outside the home. Measuring the effect of restoring some low vision to these subjects was challenging – many common activities of daily living (ADLs) and orientation and mobility (O&M) tasks could be accomplished faster and more easily without vision, or required a higher level of vision than was provided by the System. Choosing tasks that required or could benefit from very low vision therefore was a major factor that shaped the FLORA development. The team reviewed commonly accepted assessments in blindness and low vision in the areas of ADL and O&M [14-18] and identified items that had the potential to be affected by the Argus II System.

Another challenge was the spectrum of performance across subjects, even in this ultra-low vision population. We saw a range of visual function performance achieved with the Argus II System in the clinical trial, from light perception to visual acuity of 20/1260 (1.8 logMAR)[2], and experience with the subjects suggested there was a range of functional vision performance as well. The FLORA was therefore required to be able to capture functional vision performance across the spectrum of ultra-low vision.

Structure of the FLORA

Taking these guiding principles and challenges into account, the structure of the FLORA reflects a standard three-part approach of rehabilitation assessment: self-report, observation of performance, and case summary.

Part 1, the self-report part of the FLORA, includes 13 open-ended questions that were designed to guide a conversation to fully elicit the subject’s point of view on how the System has affected his or her life. For example:

• What would you like me to know about how the system has affected you?

• Has the system helped you do things you could not do before?

Self-report was viewed as a key element for the initial part of this work because we recognized that the implanted subjects were quickly becoming the experts on the use of the Argus II System, describing situations and experiences using the System that we had not considered. The self-report takes into consideration the patient’s point of view on the impact of the Argus II on their well-being.

While self-report is an important element of a comprehensive assessment, the demonstration of skills in the presence of an experienced observer is necessary to affirm or challenge the information gathered during the self-report [19]. Part 2 of the FLORA provides the evaluator the opportunity to observe the subject performing common ADLs and O&M while using the Argus II System and without using it (i.e., using only the subject’s residual vision). To choose the tasks, the team identified activities that both required vision for all or some portion of the task and had the potential to be improved with the range of vision provided by the System. Selected tasks ranged from some that could likely be performed successfully with light perception or projection only (locate a light source, find an open doorway, determine if lights are on/off), through the spectrum of ultra-low vision, to those tasks that likely require higher levels of movement, spatial, or form vision for successful completion (recognize shapes, estimate the size of an object). Activities such as reading 12-point print, identifying the color of a traffic light, or identifying individual faces require a level of vision generally beyond that afforded by the System, and were thus not included.

The list of tasks was then organized into 5 categories beginning with Body Awareness and Orientation. This category, intended to measure subjects’ general spatial orientation skills (System OFF only), was included in order to provide the evaluator with an understanding of the subject’s spatial abilities without the input of the Argus II System. It includes tasks such as “demonstrate awareness of head and eye position”, and “use directional and positional concepts (N, S, E, W, clockwise, counter-clockwise)”. The next category, Visual Orientation, evaluates the ability to use light projection and contrast to detect the location of lights, windows, and doors. The third category addresses using vision for mobility (e.g., independently cross residential streets by following the lines of a crosswalk; avoid obstacles while walking) and the fourth assesses Daily Living Tasks, with emphasis on the visual aspects of the tasks (e.g., visually locate/identify things in the bathroom; sort light from dark laundry). Blindness is known to be socially isolating[20-26], so the fifth category was added to evaluate visual tasks having to do with social interaction (e.g., visually detecting the presence of others in a room; determining the direction of movement of people walking by).

The evaluators were instructed to observe the subject performing each assessed task with the System ON and with it OFF, using as many trials as necessary to render a judgment about how difficult the task is for the subject to complete. The evaluators were encouraged to select which tasks to observe, basing the choice on the subject’s self-reported abilities (and goals) as well as his or her performance on previous tasks. Therefore, the assessment was customized for each subject.

For each task observed, the evaluators ranked the ease of performance as impossible, difficult, moderate, or easy. In addition, they estimated how much vision was used to accomplish the task: no vision, some vision, or vision only.

Part 3 is a narrative case report summarizing the findings of the interview and the observations. The purpose was to encourage the evaluator to synthesize the information into a coherent report that provides their insights into the effect of re-introducing vision to the lifestyle of this particular subject, and to identify activities where rehabilitation could improve their use of the technology. The case report documents evaluators' findings from two points of view: functional vision effects they observed, and the subject’s self-reported general observations of well-being and satisfaction (or displeasure).

Subjects

The FLORA was administered to subjects participating in the Argus II Retinal Prosthesis System clinical trial. This multi-center study is being conducted in accordance with the Declaration of Helsinki and the national regulations for medical device clinical trials in the countries in which the study is being conducted. The study has been approved by the national ministries of health in these countries and the Ethics Committees or Institutional Review Boards of participating institutions. All subjects signed informed consent to participate. The clinical trial is posted on www.clinicaltrials.gov, trial registration number NCT00407602.

The Argus II clinical trial enrolled 30 subjects with 14 from the United States and 16 from Europe. Twenty nine subjects had retinitis pigmentosa and 1 had choroideremia All subjects had long standing blindness, reaching bare light perception an average of 15.9 years prior to enrolling in the trial. The FLORA was administered on 26 subjects. One subject had been explanted and three subjects declined to participate.

There were seven evaluators (DG, MF, NT, MB, AF, MG, LF). All evaluators were qualified in the areas of rehabilitation for the blind, orientation and mobility, or occupational therapy and participated in a training session on the implementation of the FLORA. All evaluators were reimbursed for their time and expenses.

Procedures

The FLORA was administered to the 26 participating Argus II subjects between December 1, 2010 and April 8, 2011. To perform the FLORA, evaluators typically spent 3-4 hours with each subject at and/or near the subject’s home.

Analysis

The first step in determining the utility of the FLORA was to evaluate whether the interview questions and observed tasks proved to be useful and informative when applied to ultra-low vision subjects. Given the flexible nature of the assessment (i.e., evaluators were encouraged to select questions and tasks in order to customize the assessment for each subject), an analysis of the number of questions asked and tasks evaluated was performed to investigate the evaluator’s perception of the appropriateness of the tasks for the amount and type of vision afforded by the System. In addition, the number of times tasks were evaluated was correlated with several possible explanatory factors (e.g., rated difficulty) to explore possible reasons for particular task selection.

Another important aspect of task selection is whether floor or ceiling effects were noted, either in the difficultly ratings or in the ratings of how much vision was used to perform the tasks. These factors were investigated by analyzing the range of scores shown over the list of tasks.

Results

The FLORA was completed on 26 Argus II subjects using a mix of the 7 evaluators. At the time the FLORA assessments were conducted, half the subjects were at 3.3 ± 0.4 years follow-up and the other half were at 1.7 ± 0.2 years follow-up; this bimodal distribution reflects a pause in enrollment of clinical trial subjects after the first 15 implants. The second cohort of 15 subjects was implanted with a slightly modified version of the Argus II[2].

In Part 1, there were 14 suggested questions for the self-report interview. In 22 out of 26 evaluations, all 14 questions were asked and answered. In the remaining four evaluations, a total of eight questions were skipped. Only two of these were skipped more than once (in different evaluations).

In part 2, there was a total of 35 test items the evaluators could use to complete their behavioral assessment with the Argus II System ON and OFF; the additional six Body Awareness and Orientation tasks, performed only with the System OFF, are not covered here. All 35 were selected, with an average of 20 subjects being evaluated for each test item (Table 1). The range was 5 to 26 subjects. Tasks that were evaluated for all 26 subjects were locate lights in the environment, use light from windows to determine orientation, determine whether room lights are on or off, travel within home independently, visually locate people in a non-crowded setting, and determine when people walk by. The least selected items included visually locate/identify things in the bathroom (5), heat/reheat food (9) and chop/cut food (9).

Table One.

Listing of tasks from most evaluated to least evaluated

Category	Task	Number of evaluations	Percentage using vision only System On	Percentage using vision only System Off
Visual Orientation	Locate lights in the environment	26	92	44
Visual Orientation	Use light from windows to determine orientation	26	92	40
Activities of Daily Living	Determine whether room lights are on or off	26	96	56
Activities of Daily Living	Travel within home independently	26	46	12
Interacting with Others	Visually locate people in a non- crowded setting	26	46	42
Interacting with Others	Determine when people walk by	26	31	36
Visual Orientation	Use artificial light to determine orientation (different types of lights)	25	92	42
Mobility	Avoid obstacles while walking	25	70	22
Interacting with Others	Detect the approach of another person	25	40	33
Interacting with Others	Determine the direction of movement of people walking by	25	40	33
Interacting with Others	Track another person	25	72	33
Mobility	Estimate the size of an obstacle	24	86	27
Activities of Daily Living	Locate ordinary objects at various distance (familiar environment)	24	87	36
Activities of Daily Living	Visually locate a place setting on a dining table	24	96	46
Activities of Daily Living	Identify ordinary objects at various distances	24	83	36
Visual Orientation	Visually find doorways	23	96	46
Mobility	Detect curbs	23	85	29
Interacting with others	Determine direction another person is facing	23	91	30
Visual Orientation	Recognize and use shapes for orientation and environmental information (e.g., stop sign)	22	95	42
Activities of Daily Living	Identify top step/bottom step	21	52	14
Activities of Daily Living	Negotiate stairways independently	20	40	10
Interacting with Others	Visually locate people in a crowded setting	20	90	28
Mobility	Independently cross residential streets by following the lines of a crosswalk	19	65	18
Activities of Daily Living	Visually locate clothes	18	94	28
Activities of Daily Living	Sort light from dark laundry	18	89	22
Mobility	Avoid low-hanging branches, plants, head-high shelves, etc.	15	79	13
Activities of Daily Living	Visually identify food on a plate	15	87	36
Activities of Daily Living	Visually locate/identify things in the bathroom (familiar environment)	14	85	31
Activities of Daily Living	Visually find pots/pans/utensils in the kitchen	14	92	29
Activities of Daily Living	Visually locate dishes while washing	13	92	23
Activities of Daily Living	Maintain safety: falls/spills/burns	12	58	0
Visual Orientation	Use the sun to determine orientation	10	89	31
Activities of Daily Living	Cut/chop food	9	56	8
Activities of Daily Living	Heat/reheat food	9	56	10
Activities of Daily Living	Visually locate/identify things in the bathroom (unfamiliar environment)	5	75	25
Average		20	75	29

Correlations between the number of tasks evaluated and several possible explanatory factors (the order in which the tasks were listed, the percent of tasks rated easy, the percent of tasks rated possible, and the percent of tasks performed using vision only) showed no apparent trends or significant correlations (data not shown).

To evaluate whether all categories of difficulty were used, we calculated the percentage that each category was chosen by the evaluators over all subjects and tasks. With four options of impossible (33%), difficult (23%), moderate (24%) and easy (19%) we found the evaluators used all four scoring categories for System ON evaluations.

Finally, we analyzed the amount of vision that evaluators observed the subjects using to complete the various tasks when the System was ON and OFF. When the System was ON, we found that vision only was chosen for an average of 75% of evaluations over all tasks. When evaluating tasks with the System OFF, vision only was chosen for an average of 29% of evaluations over all tasks (Table 1).

Part 3 consists of written summaries of each subject within a framework of general questions. Evaluators provided a comprehensive written summary for all subjects. These summaries, while highly individualized and written by different evaluators, report on the totality of observations for each subject. As this report is focused on evaluating how Part 1 and Part 2 were used by the assessors, Part 3 is not discussed further here.

Discussion

The FLORA was developed in response to a specific need: there are no accepted, standardized functional vision or quality of life assessments that are targeted toward the kind of vision that is restored by a retinal prosthesis. The primary purpose of this report is to describe our analysis of the assessment from the point of view of learning how assessors utilized the FLORA.

Usage of the FLORA

By allowing evaluators to customize each assessment – select the questions and tasks they believed would be relevant to each particular Argus II subject – we were able to use their selections as a reflection of the appropriateness of the questions and tasks in the FLORA. The results suggest the Part 1 interview includes mostly relevant questions, with the vast majority of interviews covering all 14 questions.

Most of the Part 2 tasks also appeared to be useful and appropriate for this ultra-low vision population, with all 35 items being tested on some subjects (low of 5), and six tasks being evaluated on all 26 subjects. The results also demonstrate that we avoided both floor and ceiling effects; evaluators made use of the full rating scale, with a distribution of scores from impossible (33%) to easy (19%). No correlations or trends were found between the number of tasks evaluated and several possible explanatory factors, including the percent of task rated as easy, and the list number of each item. While this may be a result of the small N, it does suggest that evaluators were not overly influenced by the perceived difficulty of the task or the order in which it appeared on the FLORA form.

Since the purpose of the clinical trial was to evaluate the effectiveness of the Argus II as a technology to re-introduce vision to the blind, we asked the assessors to indicate when, in their judgment, vision was and was not being used to complete the tasks, as shown in Table 1. The “amount of vision used” rating in System ON conditions was skewed toward tasks in which only vision was used (75%) rather than tasks performed with additional cues such as tactile or auditory information. This was by design; our task selection was deliberately biased towards tasks that require vision to be performed accurately. As discussed above, the clinical trial subjects had established mechanisms for coping with their blindness, and many everyday tasks could be performed faster and/or more accurately without vision. Choosing these tasks but requiring that subjects perform them visually would not represent a true test of the impact of the System on subjects’ lives and it would not accurately represent how they functioned. Including only the types of tasks usually found on low-vision functional vision assessments such as the NEI-VFQ or the Activity Inventory, however, would result in a failure to measure an impact at all, which was contrary to what we were hearing from subjects about the effect of the System.

Table 1 also indicates that some visual tasks could be performed with the system OFF by an average of 29% of subjects evaluated. Since one of the criteria for acceptance into the clinical trial was residual vision of bare light perception or worse, we assume these subjects were using their remaining native vision. There are also a number of activities -- using the sun to determine orientation (sensing heat of sun); travel within home independently (familiar environment) -- which can be accomplished without vision. As a last comment on the use of vision, some tasks, specifically detecting curbs and identify top/bottom step were assessed with vision only – no long cane – but this was done by certified O&M instructors and should not be assessed by evaluators not trained in O&M.

Future Considerations

The next step in the FLORA development is to revise it to be more useful to other researchers working with ultra-low vision populations, such as subjects in clinical trials investigating other vision restoration devices and treatments. The results reported here will help guide us toward a final question and task list that can be standardized to allow direct cross-comparisons between different subjects and populations.

In addition, studies must be completed to study the reliability and validity of the task ratings if these are to be evaluated and compared across subjects and studies. The FLORA as used in the Argus II clinical trial is available for download at http://www.2-sight.eu/ee/landing-flora and we encourage interested researchers to contact us to join the discussion on the development of this tool. We believe there is a clear need in the community for such an assessment, and this need will only grow as more vision-restoration devices and treatments are studied and become available.

Conclusions

The Functional Low-vision Observer Rated Assessment was developed to evaluate functional vision and well-being in a population of subjects whose ultra-low vision had been restored by the Argus II Retinal Prosthesis System. The list of questions and tasks was found to be appropriate and the rating scale did not suffer from floor or ceiling effects. Future revisions of the assessment will expand its usefulness for other populations of restored-vision patients.