USE OF COGNITIVE INTERVIEWS IN THE DEVELOPMENT OF THE PLUS-M ITEM BANK

Abstract

Purpose

Measuring constructs such as mobility with patient-reported outcomes (PROs) can enhance clinical and scientific understanding of how health conditions, like lower limb amputation, impact patients’ lives. When developing PRO questionnaires, cognitive interviews (CIs) are used to examine if survey items are understandable, clear, and meaningful. The aim of this study was to use CIs to inform item development for the Prosthetic Limb Users Survey of Mobility (PLUS-M), a PRO that measures mobility in prosthetic limb users.

Methods

Thirty-six CIs were conducted with 30 prosthetic limb users. Each participant responded to up to 30 items from the PLUS-M candidate item set. Each item was reviewed by a minimum of five participants who differed in self-reported mobility, literacy, level of amputation, and time since amputation. Items were revised based on participant feedback and substantially revised items were re-evaluated through additional CIs.

Results

Feedback from CIs identified substantial issues in 76 of the total 156 items. These items were subsequently modified or eliminated.

Conclusion

CIs were an essential qualitative step in the development of the PLUS-M item bank and resulted in better functioning items.

INTRODUCTION

Clinicians and researchers traditionally measure rehabilitation outcomes, such as mobility following lower limb amputation, using standardized performance measures (e.g., timed tests or clinical assessments of function [1-6]). However, performance on these measures may not adequately represent all aspects of an outcome of interest (e.g., mobility) [7]. Patient-reported outcomes (PROs) are health surveys that are designed to be answered directly by the respondent and are intended to help clinicians and researchers understand how health conditions impact fundamental aspects of the respondents’ lives. PROs are well suited to assess complex health outcomes because they solicit information about the respondents’ perspectives, opinions, and/or experiences without interpretation by another individual [8]. Well-developed PROs complement performance measures when assessing important health outcomes like mobility.

A variety of PROs are available to rehabilitation clinicians and researchers to assess mobility and mobility-related constructs (e.g. balance confidence and physical function) in persons with lower limb loss. However, many of these instruments possess inherent psychometric limitations (e.g., floor and ceiling effects, poor validity or reliability, and inability to detect clinically meaningful changes) that make them less practical for use in routine clinical care or research [9]. The Prosthetic Limb Users Survey of Mobility (PLUS-M) is being developed with modern psychometric methods to mitigate these common limitations. The resulting instrument is expected to be a valid, reliable, sensitive, and brief instrument for measuring self-reported mobility in prosthetic limb users.

Cognitive interviews (CIs) are used to solicit feedback from respondents to examine if PRO items (i.e. survey questions) are understood as intended by the PRO developers [10]. Information obtained in CIs can be used to infer the cognitive thought processes participants use when they respond to items as well as to assess if questions are interpreted similarly across participants [10-15]. Interviews are purposefully designed to explore the cognitive processes used by respondents as they comprehend the question, recall information needed to answer the question, make decisions about how best to answer the question, and select an appropriate and meaningful response [10].

Evaluation of PRO items through CIs is considered standard practice in the development of psychometrically sound PRO instruments [15, 16]. CIs allow for assessment of individual items’ function early in the development process. The information obtained during CIs may then be used to modify or remove items that perform poorly prior to administration of the instrument to a large group of respondents in order to develop scoring. Benefits of cognitive interviewing include (1) identification of problems with comprehension, recall, decision, or response processes required for participants to meaningfully answer a survey question, (2) in-depth exploration of the construct/topic of interest though a semi-structured discussion of survey items with respondents, (3) detection of structural defects that may affect performance of the questionnaire (e.g. unclear layout, illogical questions), and (4) investigation of problems in the survey instructions and response options [10]. The goal of cognitive interviewing in PLUS-M development was to enhance the overall quality of the instrument.

The primary aim of this study was to inform the development of PLUS-M survey items through use of CIs with a diverse group of prosthetic limb users. Data gathered in CIs were used to assess the content validity of candidate items, identify and modify problematic items, and eliminate those items that could not be modified to function as intended. Prior to CIs, the investigators formed an advisory panel of methodological and content experts and conducted focus groups to assist in conceptualization of a PLUS-M measurement model. Following CIs, remaining and revised items were tested in over 1000 prosthetic limb users to assess item functioning. Details regarding these efforts will be provided in future publications.

METHOD

Data collection and analysis protocols used in this study were based on recent outcome measure development efforts (e.g., PROMIS and Neuro-QoL) [8, 15, 17, 18] and contemporary standards for developing PROs [19].

PARTICIPANTS

Participants were recruited from sites in Seattle, WA; Miami, FL; and Chicago, IL, through print, email, and internet postings. To be eligible for inclusion, interested persons were required to (1) be at least 18 years of age, (2) have a lower limb amputation at or below the hip and at or above the ankle, (3) regularly use a lower limb prosthesis, (4) report the ability to speak and read English, and (5) allow the interview to be audio recorded.

In addition to the criteria listed above, purposive sampling techniques were used to ensure that each question was evaluated by at least five participants who collectively possessed the following characteristics: (1) self-report of high mobility, (2) self-report of fair-to-low mobility, (3) lower than grade 12 education or self-report of cognitive impairment, (4) bilateral lower limb amputation, (5) amputation above the knee, and (6) less than one year since their amputation. Single participants could represent more than one of these traits (e.g. self-report of high mobility and bilateral lower limb amputation) as the intent of the described sampling procedure was to include a wide range of perspectives and experiences. Additional CIs were conducted for items that were modified as well as those that targeted specific populations (e.g. prosthetic limb users who participate in sport activities) to ensure that items are reviewed by people who have experienced the activity described in the item.

The sample size was deemed sufficient as CIs in this study were used to qualitatively assess individual items prior to their inclusion in the PLUS-M item bank, not to derive statistical estimations [14]. In qualitative research (e.g. CIs), the sample must be sufficient to elicit a variety of the perspectives that are relevant to the construct of interest (i.e. mobility with a prosthetic limb). That is, the objective of qualitative research designs is to reduce the chances of discovery failure rather than recruit a sample that is representative of the population [16]. In addition, many of the items tested were modifications of existing items that had presumably been tested by respondents during development of their parent instruments. Similar sampling protocols have been deemed acceptable in previous standardized instrument development efforts [8, 15, 17] and small sample sizes are recognized as an inherent attribute of cognitive interviewing [14].

The study was approved by the University of Washington Institutional Review Board. All participants gave informed consent prior to the interview.

INITIAL PLUS-M SURVEY

The investigators conducted a thorough literature review to identify self-report instruments of mobility or mobility-related constructs (e.g. physical function). More than 1000 candidate items from 41 unique instruments were identified. Many items addressed similar concepts (e.g., walking up stairs) and were therefore subjected to a process of binning and winnowing [15]. New items were then created by investigators based on professional experience and feedback from focus groups held with prosthetic limb users. Ultimately, 130 items were considered for inclusion in the initial PLUS-M survey. Following initial rounds of CIs, twenty-six additional items were added based on participants’ suggested additions and/or revisions. Thus, a total of 156 items were tested with CIs. Most candidate items were modified or created to share the following general structure: they began with the context “Are you able to” and the available response options were “without any difficulty”, “with a little difficulty”, “with some difficulty”, “with much difficulty”, and “unable to do”. The exceptions included one item that assessed confidence (“How confident are you”) and another item that assessed frequency (“How often do you”). The initial survey instructions and examples of candidate items can be found in Tables 1 and 2.

Table 1.

PLUS-M Instructions

Original instructions:

We want to know how well you can move around using your prosthetic leg. Please respond to all questions as if you were wearing the prosthesis you would normally use to perform the task. If you would normally use a device that helps you walk or balance (e.g. a cane, crutch, or walker) while performing the task, please answer the questions as if you were using that device. Please do not answer questions as if you are sitting in a wheelchair or receiving support from another person.

Final instructions:

Please respond to all questions as if you were wearing the prosthetic leg you use most days. If you would normally use a cane, crutch, or walker to perform the task, please answer the questions as if you were using that device. Please choose “unable to do” if you: 1)Would need help from another person to complete the task 2)Would need a wheelchair or scooter to complete the task 3)Feel the task may be unsafe for you

Table 2.

Sample Questions and Examples of the Revision Process

Type of Problem	Original Question “Are you able to...”	Decision Made	Revised Item “Are you able to...”
Unclear intent	... walk on a sideways incline (e.g. a sidewalk that slopes toward the street)?	Item revised to clarify the surface characteristics intended by the developers.	... walk on a surface that slants sideways where one side is higher than the other?
	... walk in a narrow hallway?	Item was revised. Developers intended the question to reflect a situation where the respondent would need to narrow their base of support, which is not how respondents interpreted the question.	... walk with your feet close together.
Inappropriate assumptions	... open and close windows in your home?	Item was deleted. Respondents reported that the item reflected window function rather than mobility.	N/A
	... stand up on tiptoes?	Item was deleted. Respondents reported that standing on tiptoes is not appropriate for persons who have typical prosthetic feet.	N/A
Double-barreled	... walk when you are unable to see your feet or the ground (e.g. deep leaves or tall grass)?	Item was split into two independent items. Respondents reported that deep leaves were easier to walk in than tall grass.	... walk without looking down at your feet? ... walk in tall grass (e.g. a field)?
	... walk on slippery surfaces (e.g. wet tile, ice, a rainy street, or a wet deck)?	Item was revised. Respondents felt that the examples given differed in their difficulty level.	... walk on slippery surfaces (e.g. wet tile or ice)?
No problems identified	... walk on hardwood floors?	No changes made.	N/A

N/A = not applicable

PROCEDURE

CIs were conducted in person (n=8) or via telephone (n=28) by study investigators to accommodate involvement by both local and national participants. The protocol was the same for both types of interviews. Participants were first asked to complete a hard-copy of the initial survey. The survey was either mailed to participants with instructions to complete the survey at the time of interview (for telephone interviews) or provided to them at the interview (for in-person interviews). Immediately following completion of the survey, the investigators asked open-ended follow-up questions about each survey item. Thus, the time between completing the written survey and beginning the interview process was generally less than 5 minutes. The interview process required, on average, 56 minutes (standard deviation = 23 minutes) to complete. Due to the large number of candidate items (n=156), they were divided into seven sets, each with between 25-30 items. Each set included items that spanned a range of mobility activities (i.e. not all items were related to a single topic, such as stair climbing). In the process of conducting interviews, the investigators noted that a number of questions (e.g. “Are you able to run 10 miles?”) measured athletic activities that most respondents had never experienced. Therefore, two additional prosthetic users who regularly participated in athletic activities were recruited to review these questions. Participants also responded to questions related to the respondent's general health, activity level, prosthesis, and cognitive function.

Three investigators were trained by an experienced qualitative researcher to conduct CIs using retrospective verbal probing. In this approach, the participant is asked to fill out the survey without any guidance from the interviewer just prior to an item-by-item oral interview. Verbal probing is then used to solicit information about the thought processes used by the participant to answer each question. Willis recommends retrospective verbal probing when evaluating written surveys because it allows the respondent to complete the measure in the intended manner (i.e., unaided, as if in a medical office), thus minimizing the effect of the interview on the respondent's choices [14]. In this study, the number of items administered at each interview was restricted (i.e., each respondent was given 30 or fewer survey questions) to allow for more accurate recall by participants and to limit the total time required for the interview. Retrospective verbal probing has also been adopted by large national initiatives for the development of recent self-report measures (e.g., PROMIS and Neuro-QoL) [8, 15, 17].

Interview guides were compiled for each interviewer in order to standardize the CI process across investigators. Each guide contained the subset of the items that were to be administered to the participant and an associated list of scripted questions (i.e. probes) that were to be used for the oral interview. These probes were designed to assess comprehension/ interpretation (e.g. “what does the term ‘transfer’ mean to you?”) and recall (e.g. “when was the last time you walked continuously for 15 minutes?”) as well as to solicit general information about the participant's cognitive process when answering the item (e.g. “tell me what you were thinking when you answered that question”) [10]. Interviewers also asked unscripted questions (i.e. spontaneous probes) when appropriate to better understand the participant's response. The interviewing team included two researchers who are certified and licensed prosthetists and one researcher who is a prosthetic limb user.

Interviewers audio-recorded all interviews and took field notes to supplement the recordings. Interviewers compiled a detailed summary of each interview within one day of the interview, referencing the audio recording and/or field notes as needed. Interviewers marked each item as “no change” if the item appeared to function as intended or “change and/or learn more” if the item did not function as intended. Investigators discussed items marked as “change and/or learn more” during item review.

ITEM REVIEW

Following the first round of CIs (n≥5 participants per item), interviewer summaries were organized into a single document that detailed collective participants’ feedback for each item. Investigators, as a group, reviewed feedback across respondents and discussed potential concerns with each item. Group consensus determined how items were managed. If the item did not function as intended, the item was revised to address concerns identified in the CIs.

Items that were substantially revised were re-reviewed in a second round of CIs. ‘Substantial revision’ was defined as any change to the item that involved more than (1) addition or removal a supportive word, (2) addition or removal of a word that did not change the meaning of the phrase, (3) substitution of a word to simplify the item semantics, or (4) a change to the order of the words in the item. Revised items were administered to participants who provided useful feedback in the first round of CIs (n=3 participants per item). In addition, a third round of CIs was performed with two athletic participants to further assess well functioning items from the first round or revised items from the second round that involved running or participation in sport activities. Finally, a fourth round of CIs (n=5 participants per item) was performed to assess new items that were constructed by the investigators following feedback given in the first two rounds of CIs. Second-, third-, and fourth-round CIs were conducted in the same manner as the initial CIs. Group consensus by the study investigators was again used to decide if the item was to be revised or if it should be removed from inclusion in the candidate item bank. Improvement in revised items was gauged by participants’ ability to easily understand the intent of each item and provide a meaningful response.

Items that did not have identified concerns and items that were successfully revised were compiled into a candidate item bank to be tested with a large sample of prosthetic limb users.

RESULTS

Thirty participants were involved in a total of 36 CIs. Participants were diverse with respect to current age, age at amputation, years since limb loss, and etiology of amputation. About two-thirds of the participants were male. This is consistent with the gender ratio of persons with lower limb loss reported in the literature [21, 22]. Approximately half of the participants were from Washington State and half from other locations in the United States. Demographic characteristics of the study sample are provided in Table 3.

Table 3.

Participant Characteristics

Participant Characteristics (n=30)		Mean (SD)	Range
Age (yrs)		50.2 (13.9)	(26-77)
Years since first lower limb loss (yrs)		14.9 (13.9)	(0.4-52.3)
Prosthetics Evaluation Questionnaire – Mobility Subscale (PEQ-MS)*		2.9 (0.9)	(0.8-4.0)
		N	Percent
Gender	Female	10	33.3%
	Male	20	66.7%
Sample characteristics	Bilateral	10	33.3%
	Unilateral	20	66.7%
	Transfemoral	14	46.7%
	Transtibial	17	56.7%
Race/Ethnicity	Non-Hispanic White	18	60.0%
	Non-Hispanic Black	8	26.7%
	Hispanic	3	10.0%
Education	Some High School	6	20.0%
	High School Grad/GED	5	16.7%
	Some College	7	23.3%
	College Degree	6	20.0%
	Advanced Degree	4	13.3%
Employment Status	Employed	9	30.0%
	Retired	6	20.0%
	On disability	6	20.0%
	Unemployed	4	13.3%
	Homemaker	2	6.7%
	Other	2	6.7%
Cause of Limb Loss (May select multiple)	Trauma	15	50.0%
	Infection	10	33.3%
	Diabetes	6	20.0%
	Other	7	23.3%
Participant Location	Washington	16	53.3%
	Illinois	8	26.7%
	Alaska	1	3.3%
	Arizona	1	3.3%
	Florida	1	3.3%
	Ohio	1	3.3%
	Utah	1	3.3%

^*Prosthetics Evaluation Questionnaire score (PEQ-MS) has a possible range of 0-4; higher scores indicate better functioning [20].

The quality of PLUS-M items, instructions, and response options were examined and improved based on participant feedback. A total of 205 items were examined across four rounds of CIs (Figure 1). This included both the original 156 items that were tested in the first round of CIs and those items (e.g. alternate versions of original items) that underwent additional rounds of testing. Of the 156 original items, 80 (51%) were not substantially revised, 22 (14%) were substantially revised, and 54 (35%) were deleted based on CI feedback. It should be noted that although 80 items were not substantially revised, minor edits were made when necessary to simplify the language of the items. The resulting 105 items were included in the candidate PLUS-M item bank. The item revision process is described below and examples are included in Table 2.

Fig. 1.

Schematic depicting the CI process used in the development of the PLUS-M candidate item bank

INSTRUCTIONS

Participants were asked about the usefulness of the instructions. Many participants reported that they did not carefully read the instructions prior to answering the PLUS-M questionnaire. Those that reviewed them often stated that the instructions were “good”, “helpful”, and “made sense.” Suggestions for improvement included shortening the instructions and addressing concerns of participants who may view the described activity (e.g., walking while carrying a small child) as unsafe. In response to this feedback, the instructions were simplified and formatted with bullet points to reduce respondent burden. In addition, language was added instructing respondents to choose “unable to do” for situations where safety concerns would restrict engaging in that activity.

RESPONSE OPTIONS

Participants were also asked to comment on the appropriateness of the response options. Most participants made comments about the response options, such as “they were reasonable” or “I didn’t notice any problems.” However, one participant suggested that respondents be asked if they had performed the activity and another stated that the addition of “not applicable” response option would be helpful to accommodate for activities that the respondent had not tried. It is understandable why respondents wished to have a “not applicable” or “do not know” response available. However, including such options in a survey instrument introduce issues related to scoring and do not enhance the quality of data [23]. For these reasons, “not applicable” and “do not know” response options were not added.

CANDIDATE ITEMS

Participants provided detailed feedback on problematic candidate items. Common problems seen in items included issues with clarity, inappropriate assumptions, and double-barreled items. These problems are discussed in further detail below.

Item intent is unclear

Problems with item clarity were most commonly identified in PLUS-M candidate items. In such situations, respondents perceived items differently than what was originally intended by the investigators. For example, “Are you able to walk on a sideways incline (e.g. a sidewalk that slopes toward the street)?” was understood by respondents as a question about walking on a ramp or a curb cutout. However, the question was intended to assess walking on a surface when the feet are at unequal heights. Because the original item was so challenging to understand, it was rewritten as, “Are you able to walk on a surface that slants sideways where one side is higher than the other?” Subsequent rounds of CIs showed the intent of the revised item was more clearly understood by survey respondents.

Item assumptions are inappropriate.

Another problem identified in candidate items was an implied assumption that experiences related to the item content were similar across respondents. In reality, experiences differed in ways that markedly affected participants’ responses to some questions. For example, participants’ responses to the item, “Are you able to open and close windows in your home?” varied based on the type and condition of windows in their home. Some respondents replied that this task was quite difficult because their windows stuck, while others found it easier because they had newer windows. As such, responses were strongly influenced by the respondents’ experiences, rather than their mobility. This question was therefore deleted.

Items are double-barreled

Some items were identified as problematic because they were double-barreled, (i.e. they combined two different questions). Double-barreled items were challenging for participants to answer because they wished to respond differently to different parts of the item. For example, respondents reported challenges responding to the item “Are you able to walk when you are unable to see your feet or the ground (e.g. deep leaves or tall grass)?” They indicated that it was more difficult to walk in tall grass than it was to walk in deep leaves. Although both tall grass and deep leaves were reported to restrict view of the respondents’ feet, tall grass was noted to resist swing phase of the prosthetic limb far more than did deep leaves. Investigators therefore chose to split this question into two items, one that specifically addressed respondents’ ability to walk without looking at their feet and another that addressed their ability to walk when motion of the prosthesis was inhibited.

APPROPRIATENESS OF ITEMS FOR SUBPOPULATIONS OF PERSONS WITH LOWER LIMB LOSS

Persons with less than one year of experience

New prosthetic limb users varied in their responses to candidate PLUS-M items based on the amount of experience they had using a prosthesis. Two participants with between 0.7 years and 1.0 years experience easily related to the scenarios asked of them and thus had no difficulty responding to survey items. Two other participants with limited prosthetic experience (i.e. each with 0.3 years) reported that they had not experienced many of the activities and could not provide meaningful answers. One of these individuals stated multiple times that she would prefer to answer “not applicable” because she had never tried the activities. These two participants often envisioned performing the activities while sitting in a wheelchair, rather than while walking with their prosthetic limb. This suggested that activities performed recently after amputation required the use of wheeled assistive devices and that responses to items did not reflect the developers’ intent of measuring prosthetic mobility.

Persons with upper extremity involvement

Two individuals in this sample noted that their upper extremity involvement (i.e. injury or amputation of the upper limbs) affected the way that they answered PLUS-M items. One person with shoulder dysfunction stated that his shoulder injury, rather than his amputation, made “playing catch with a ball” difficult. Another person with upper limb loss noted that the difficulty that he reported “weeding a garden” was due to his upper limb loss and not mobility problems related to use of his lower limb prostheses. This feedback suggested that some items functioned differently in persons with lower limb loss and upper extremity involvement than they did in those with lower limb loss alone.

Persons with bilateral amputation

Prosthetic limb users with bilateral amputation also provided feedback about all candidate items during CIs. Interestingly, PLUS-M items were understood similarly both by participants with bilateral amputation and participants with unilateral amputation.

Persons with low literacy

The candidate items were tested in persons with either self-reported cognitive dysfunction and/or by persons who did not graduate from high school or pass the general education development (GED) exam. In general, items were read and understood by those in this group. One individual with self-reported cognitive dysfunction took longer to read and process items, which was reflected in a longer CI time (110 minutes) than other participants (16-94 minutes). However, once he finished reading the questions, he correctly understood their intent and discussed his thought process for deriving an answer with the interviewer. Although CI results suggested that the PLUS-M items were comprehended by persons with low literacy, the investigators also modified items whose Lexile® score was higher than 1100L to ensure that they would be understood by those at an eighth grade reading level [24].

DISCUSSION

CIs resulted in content modifications that substantially improved the clarity, comprehensibility, and quality of PLUS-M items. Problems identified in CIs were addressed through revision and/or deletion of items that could not be modified to function as intended by the PLUS-M investigators. Results of this study suggest that minor changes in the wording and format of the items, instructions, and response options can substantially change respondents’ interpretation of item content. Addressing problems identified by respondents (e.g. items that are unclear, make inappropriate assumptions, or are double-barreled) is an essential and useful step in instrument development. The importance of CIs is indicated by Federal Drug Administration (FDA) patient reported outcomes (PRO) guidelines that require documentation of content validity through qualitative studies with members of target respondents [25]. Data that show instrument items are understandable and appropriate for the intended population and use are considered evidence of content validity [26]. The FDA guidance therefore recommends use of CIs to evaluate respondents’ understanding of the questions, instructions, and response options [25].

The CIs administered in this study also provided information about the usefulness of items for subpopulations within lower prosthetic limb users. Some items appeared to function differently for persons with limited prosthetic experience and for persons with upper extremity involvement. Persons with limited prosthetic experience reported that they have not been exposed to many of the activities described in the candidate items and therefore could not meaningfully respond. Persons with upper extremity involvement noted that answers to items that inquired about mobility tasks that included upper limb use were mainly driven by their upper limb concerns and not their lower limb mobility. Future research is therefore warranted to better understand the function of PLUS-M items in persons with limited prosthetic experience or upper extremity involvement. Until more evidence is available, use of this instrument in these populations should be approached with caution. The candidate items functioned appropriately for persons with bilateral lower limb loss, though additional research is needed to ensure the appropriateness of the PLUS-M for those in this population. Lastly, candidate PLUS-M items are appropriate for respondents with low literacy. However, the administration of the survey to persons with lower literacy may take longer than is typically expected and Lexile® readers below 1100L may experience challenges in comprehension with some items.

This summary of use of CIs to inform development of the PLUS-M item bank represents part of a larger project. This project to-date has included development of a conceptual framework, review of existing measures of mobility and physical function, and conduct of focus groups of prosthetic limb users (details regarding those methods will be published elsewhere). In addition, following CIs, the modified items were administered to a group of over 1000 prosthetic limb users to further evaluate the function of the items and to develop scoring for the PLUS-M.

LIMITATIONS

Thirty individuals with limb loss participated in the cognitive interviewing process, but only about five participants reviewed each individual item. Five participants cannot provide all perspectives and interpretations. However, the goal of this qualitative step was to better understand how the items function when reviewed by individuals with a variety of backgrounds and experiences [8]. It is more important at this step in item development to have a range of experiences represented through a diverse sample than a large number of people [14-16]. Nevertheless, it is possible that CIs with additional respondents would have identified problems not identified by the participants in this study.

Study participants included individuals the investigators considered representative of persons with low literacy. Low literacy in this study was established via participants’ self-report of educational experience and/or difficulties with thinking, memory or concentration. The investigators acknowledge that direct testing of literacy through established measures such as the Wide Range Achievement Test (WRAT) [27] or the Rapid Estimate of Adult Literacy in Medicine (REALM) [28] may provide more accurate assessments of literacy. Cognitive testing of PLUS-M items with persons deemed to be of low literacy based on scores from these or similar instruments should be considered in future investigations to verify that the survey is suitable for persons with low literacy.

Another potential limitation common to all CIs that use retrospective probing is participant recall [10]. A participant may not remember how they arrived at their response if follow-up questions are presented long after they respond to the tested item. To minimize this limitation, we chose to administer few items (i.e., 25-30) and held the interviews immediately after the participant filled out the questionnaire. These steps reduced the time between completing the questionnaire and answering the interview questions, thus increasing the likelihood that participants were able to remember and comment on their experience when responding to the item.

The study investigators recognize that CIs are only one step towards establishing evidence of a measure's content validity. Thus, additional work has been undertaken (e.g., advisory panel reviews, conceptual framework development, construct definition, focus groups, and literature review) but is not detailed in this manuscript. This work will be described in future publications regarding development of the PLUS-M instrument.

CONCLUSION

CIs were effective in identifying and resolving concerns with candidate PLUS-M items. Participant feedback resulted in considerable improvement of the measure and resulted in 105 well-functioning items for measuring self-reported mobility of lower limb prosthetic users. Although engagement in item development processes like CIs was time- and resource- consuming, taking these steps ensure that PLUS-M will be a valid and useful tool to measure mobility in prosthetic limb users. CIs substantially improved the relevance of the instrument to intended users, and ultimately make the measure more clinically meaningful.