The Development and Testing of a Module on Child Functioning for Identifying Children with Disabilities on Surveys. II: Question Development and Pretesting

Abstract

Background.

This paper is part of a series of articles documenting the development of a module on child functioning by UNICEF in collaboration with the Washington Group on Disability Statistics (WG). This paper documents the contribution of Cognitive Interview (CI) question evaluation methods to the development of the final module.

Objective.

The overall goal of this project was to develop a cross-nationally comparable module to measure child function and disability. Specifically, the goals of the question evaluation study were to investigate question interpretation, sources of error and bias and to use the results iteratively in the development of the final module.

Methods.

As is standard in CI studies, data were gathered through one-one-one, in depth interviews. A total of four rounds of testing, comprising 385 Cognitive Interviews, were conducted across six countries. Qualitative data analysis methods were used to identify patterns of question interpretation and areas potential error and bias among sub-groups of respondents.

Results.

Through an iterative process of testing and revision, analytic findings from these interviews were used to guide decisions on question inclusion, revision and deletion. Four types of revisions were made: 1) changing, deleting or adding specific words; 2) moving, deleting or adding clarifying phrases; 3) revising or deleting items for conceptual clarity; and 4) adding examples.

Conclusions.

These efforts to reduce error and bias resulted in a validated module that can provide cross-nationally comparable measures of child functioning.

Introduction

Measurement of the complex phenomenon of disability is even more complicated in a multi-national context. Linguistic, socio-cultural and economic differences can have a profound effect on how concepts related to disability are understood. However, a key goal of UNICEF in collaboration with the Washington Group on Disability Statistics (WG) was to develop a survey module on child functioning that could be used to collect comparable data cross-nationally. An important part of the process of developing the final module was the evaluation of the proposed questions. A primary goal of question evaluation is to determine whether survey items adequately capture constructs as they were intended by the survey designers. In the measurement of disability, it is first necessary for survey designers to disentangle the complex phenomena which underlie each individual domain of disability and to formulate questions that focus on these discrete domains. Survey responses that capture constructs other than those intended are known as measurement error.

A further goal of question evaluation is to systematically examine the comparability of constructs captured across sub-groups of respondents. In practical terms, this means determining whether these constructs are understood similarly, for example, by respondents in different countries, respondents who speak different languages or respondents with differing levels of disability or functioning. When constructs are understood differently by different groups, this can lead to response bias. Once potential sources of measurement error and response bias have been identified, revisions can be made to reduce or eliminate error and bias.

Cognitive interviewing (CI) is recognized as a valuable method of evaluating surveys in multi-national, multicultural contexts.^1,2,3 CI studies can identify overall patterns of interpretation for each question and potential sources of measurement error and bias. As a qualitative methodology, CI provides insight into the broader context in which respondents answer questions. CI illuminates not only sources of error and bias but also the details of respondents’ lived experiences that lead to error and bias. Deep understanding of the real-life context in which questions operate can lead to effective revisions. Further, elucidation of the ways that respondents understand the constructs that underlie questions provides evidence of overall question validity. ⁴

Utilizing CI methodology, the main goals of this project were to: 1) assess respondents’ interpretations of the survey questions, 2) identify potential for error due to question wording 3) identify potential sources of bias by examining patterns of interpretation and patterns of error across subgroups of respondents, and 4) develop a finalized module through iterative rounds of testing and revision.

Materials and Methods

CI Procedure

In-depth, one-on-one, semi-structured interviews with a small, purposive sample of respondents typically provide the raw data of the cognitive interview. Respondents are recruited based on criteria relevant to the topic of the instrument being tested. During each interview, a trained interviewer administers the instrument to be tested, ascertains the respondent’s answer to each question and uses verbal probes to understand why the respondent provided those particular answers. Verbal probing can occur retrospectively (after all questions have been asked) or concurrently (after each question). The interviews are semi-structured in that although interviewers may probe on certain predetermined aspects of the questions (such as particular terms or the timeframe reference periods), probes are generally open ended, allowing for detailed and expansive exploration of the context of the respondent’s understanding of the phenomena captured by the questions.

As is typical in qualitative analysis, the raw data provided by the interviews is subjected to an iterative process of synthesis and reduction.⁵ The process begins with a large amount of data (the interviews and interview transcripts) and ends with a summary of observed patterns relevant to the study goals. There are five stages of analysis: conducting the interview; summarizing the interview; comparing data across respondents; comparing data across subgroups of respondents and making final conclusions.⁴ In each of these stages, data are further reduced until the final stage when concise conclusions about a question’s performance are presented.

Study Protocol

The evaluation of WG questions was based on 385 interviews that were conducted by research teams with the guidance of the Collaborating Center for Questionnaire Evaluation and Research (CCQDER) at the National Center for Health Statistics (NCHS). Because this project involved multiple rounds in multiple counties, the sample size was large relative to most CI projects. For questions intended to measure disability, parents of children both with and without disabilities were recruited. Interviews were conducted in six countries in four separate rounds of testing. The first round of testing, consisting of 217 interviews, was conducted in the United States, India, Oman, Belize and Montenegro. Questions were modified based on findings of round one. In round two, these revised questions were tested in 25 interviews conducted in the United States. Round three, consisting of 16 interviews and also conducted in the United States, tested minor revisions based on the findings of round two. A final round of testing, with a total of 127 interviews, was conducted in India and Jamaica.

Interviewer training:

All stages of the project were overseen by CCQDER/NCHS staff. Interviews in the United States were conducted by researchers from CCQDER/NCHS. For interviews conducted outside of the United States, CCQDER/NCHS partnered with organizations in each country including ADAPT in India, The National Center for Statistics and Information in Oman and UNICEF country offices in Jamaica, Belize and Montenegro. CCQDER/NCHS staff provided extensive, in-country interviewer training to prepare interviewers from the partnering organizations to conduct interviews. In-country recruiting and interviewing was completed by staff from the partnering organizations under the direction of CCQDER/NCHS.

Sample.

Staff at CCQDER/NCHS and partnering organizations in each country worked to identify potential respondents. From these pools of potential respondents, a purposive sample was recruited of parents or guardians of children ages 2 to 17 who either had no functional difficulties or who had difficulties in one or more of the following areas: seeing, hearing, walking, learning, and/or behavior. Respondent demographics for the full sample are shown in Table 1. The sample (those interviewed) was a little over half female, and most of the interviews were conducted in English.

Table 1.

Demographic profile of respondents

		N=384	Total (%)
Gender
	Female	239	62%
	Male	145	38%
Country
	Belize	45	12%
	India	134	35%
	Montenegro	45	12%
	Oman	32	8%
	United States	63	16%
	Jamaica	65	17%
Language of interview
	English	242	63%
	Arabic	33	8%
	Hindi	64	17%
	Montenegrin	45	12%
Age of child
	2–4	77	20%
	5–8	146	38%
	9–11	58	15%
	12–14	52	14%
	15–17	51	13%

Interviewing procedures.

The questions were written in English and then translated into Hindi, Montenegrin and Arabic. Interviews were conducted by native speakers of each language. During the interviews, retrospective, intensive verbal probing was used to collect detailed information on how respondents formulated their answers. First, respondents were administered all questions, and then interviewers returned to each question and probed retrospectively. Probes included: Why did you answer the way that you did? How did you arrive at your response? Can you tell me more about that? Can you clarify what you mean? Video or audio recordings were made and written notes of interview summaries were compiled. All interviews were conducted face-to-face. Interviews typically lasted 60 minutes and respondents were remunerated for their time.

Analysis.

Summary notes were written in the language in which the interview was conducted and, if necessary, translated into English. Analysis was conducted in English by trained researchers at CCQDER/NCHS. Analysts used Q-Notes, a data analysis software tool. Q-Notes was developed by CCQDER/NCHS specifically for the purpose of facilitating analysis of large, multi-lingual, multinational CI projects.⁶ Q-Notes was first developed and used in the testing of WG questions on adult disability. ⁴

Revisions.

The process of testing and revising was iterative and exhaustive. After each round of testing, vague, confusing or ambiguous questions were revised. Questions were tested, improved and then tested again until a final set of questions were identified which adequately captured intended constructs. Throughout the process, the constructs were refined. Redundant questions were deleted as necessary.

Results

Addressing Error:

In order to reduce potential sources of error, analytical findings were used to make changes to the overall instrument and to specific questions. Almost all of the original items were either revised or deleted based on results of the cognitive testing. Four basic types of revisions were made. These types of revisions included: 1) changing, deleting or adding specific words; 2) moving, deleting or adding clarifying phrases; 3) revising or deleting items for conceptual clarity; and 4) adding examples. Full revisions are detailed in Table 2 with explanatory comments and examples given below.

Table 2.

Item revisions

Domain	Purpose	Original item*	Final revision	Type of revision	Reason for revision
Seeing	Identify children who have seeing difficulties. Seeing difficulties includes problems seeing things in day or night, close up or far away, having reduced ability to see out of one or both eyes, having limited peripheral vision.	Does [name] wear glasses? Yes/no	Does (name) wear glasses or contact lenses?	Add word or phrase	Needed for clarity
		Does [name] have difficulty seeing [when wearing his/her glasses]?	[When wearing his/her glasses or contact lenses], does (name) have difficulty seeing?1	Move clarifying phrase	To encourage respondents to attend to inclusion criteria for item
Hearing	Identify children who have hearing loss or auditory problems of any kind. This includes reduced hearing in one or both ears, the inability to hear in a noisy environment or to distinguish sounds from different sources. The question is not intended to capture children who can hear the sounds but either do not understand or choose to ignore what is being said to them. Those concepts are captured in the communication domain.	Does [name] use a hearing aid? Yes/no	No revision
		Does [name] have difficulty hearing [when using his/her hearing aid(s)]?	[When using his/her hearing aid], does (name) have difficulty hearing sounds like peoples’ voices or music?	-Move clarifying phrase -conceptual revision -Examples added	To clear up conceptual ambiguity and to encourage respondents to attend to inclusion criteria for item
Walking	Identify children with varying degrees of gross motor difficulties. Walking is the primary mode of how people move around and cover distances without the use of assistive devices. Walking is a good measure of gross motor skills because it requires a mix of strength, balance, and the ability to control body movements against gravity. Difficulty walking can also be caused by difficulties with seeing.		Does (NAME) use any equipment or receive assistance for walking?	-Item added	to determine inclusion criteria/skip patterns for following items
		(2–4)Compared with children of the same age, does [name] have difficulty walking?	[with/without his/her equipment or assistance], does (NAME) have difficulty walking?	-Add clarifying phrase	To encourage respondents to attend to inclusion criteria for item
		Compared with children of the same age, does [name] have difficulty walking 100 meters on level ground?	[[with/without] his/her equipment or assistance /Compared with children of the same age], does (name) have difficulty walking 100 yards/meters on level ground? That would be about the length of 1 football field. [Or insert country specific example].	-Add clarifying phrase -Example added	To encourage respondents to attend to inclusion criteria for item and for item clarity
		Compared with children of the same age, does [name] have difficulty walking 500 meters on level ground?	[[with/without] his/her equipment or assistance /Compared with children of the same age], does (name) have difficulty walking 500 yards/meters on level ground? That would be about the length of 5 football fields. [Or insert country specific example].	-Add clarifying phrase -Example added	To encourage respondents to attend to inclusion criteria for item and for item clarity
Self-care	Identify children who have difficulty taking care of themselves as the result of functional difficulties in any domain (seeing, cognition, walking, etc.).	Compared with children of the same age, does [name] have difficulty with self-care such as feeding or dressing him/herself?	Does (name) have difficulty with self-care such as feeding or dressing him/herself?	-remove comparison phrase	To simplify the item
Fine Motor	Identify children with difficulty in the coordination of small muscle movements (i.e., fine motor difficulties)		(2–4) Compared with children of the same age, does (name) have difficulty picking up small objects with his/her hand?†	-question added	This difficulty was not covered by any other domain
Communication	Identify children who have difficulty exchanging information or ideas with others at home, school or in the community through the use of spoken language. If a child does not have spoken language and does not have an available accommodation it will be very difficult for him or her to communicate, particularly outside of the immediate family.	(2–4) Does [name] have difficulty understanding you?	(2–4) Does [name] have difficulty understanding you?	-No revision
		(2–4) Do you have difficulty understanding what your child wants?	When (NAME) speaks, do you have difficulty understanding him/her?	-add clarifying phrase	Clarify item intent
		Compared with children of the same age and using [his/her] usual language, does [name] have difficulty understanding other people?		-delete item	To eliminate redundancy
		Compared with children of the same age and using [his/her] usual language, does [name] have difficulty being understood by other people?	When (name) speaks, does he/she have difficulty being understood by people inside of this household?	-Split into two separate items -Add clarifying phrase	To clear up conceptual ambiguity
			When (name) speaks, does he/she have difficulty being understood by people outside of this household?
		Can your child speak at all? ‡		-delete item	Item did not effectively screen out non-verbal children
Learning and cognition	Identify children with cognitive difficulties that make it difficult to learn new information, language, concepts and skills. All aspects of learning are included. The information or skills learned could be used for school or for play or any other activity.	(2–4) Compared with children of the same age, does [name] have difficulty learning the names of common objects?	Compared with children of the same age, does (name) have difficulty learning things?	-Delete word or phrase	To simplify the item
		Compared with children of the same age, does [name] have difficulty learning to do new things?
		Compared with children of the same age, does (name) have difficulty remembering things?	Compared with children of the same age, does (name) have difficulty remembering things?	-No revision
Emotions	Identify children having difficulties expressing and managing emotions. All children have some worries and may feel sad, but when these worries result in the child being restless, tired, inattentive, irritable, tense, and having sleep problems, they may interfere with the child’s schooling and social development.	Compared with children of the same age, how much does (he /she) worry or feel sad? 1)Less/Not at all 2) The same 3) More 4) A lot more	How often does (name) seem very anxious, nervous or worried? Would you say: daily, weekly, monthly, a few times a year or never?	-Split into two separate items -added word or phrase	To clear up conceptual ambiguity and to eliminate inclusion of “normal” levels sadness or worry
			How often does (name) seem very sad or depressed? Would you say: daily, weekly, monthly, a few times a year or never?
Behavior	Identify children with behavioral difficulties that limit their ability to interact with other people in an appropriate manner. For young children this can include kicking, biting and hitting. For older children this can include telling lies, fighting, bullying, running away from home, or skipping school/playing truant.	(2–4) Compared with children of the same age, how much does (name) kick, bite or hit other children or adults? 1) The same or less 2) More 3) A lot more	Compared with children of the same age, how much does (NAME) kick, bite or hit other children or adults? Would you say: not at all, the same or less, more or a lot more?	-Response options revised	To align with respondent response process
		Compared with children of the same age, how much difficulty does (name) have controlling his/her behavior? The same or less, More, A lot more	Compared with children of the same age, does (name) have difficulty controlling his/her behaviour? no difficulty, some difficulty, a lot of difficulty or cannot do at all?	-Item reformatted	To align with other items in the set
Attention	Identify children with attention difficulties that limit their ability to learn, interact with others and participate in their community.	Compared with children of the same age, does (name) have difficulty concentrating on a task?	Does (name) have difficulty concentrating on an activity that he/she enjoys doing?	-revise word or phrase	clarity
Coping with Change	Identify children with cognitive or emotional difficulties that make them very resistant to change.	Compared with children of the same age, does (name) have difficulty accepting change to plans or routine?	Does (name) have difficulty accepting changes in his/her routine?	-Revise word or phrase	To simplify and clarify the item
Playing	Identify children with a difficulty in playing that is related to any functional difficulty, as difficulty in any domain can affect playing, depending on the environment.	(2–4)Compared with children of the same age, does [name] have difficulty playing with toys or household objects?	(2–4) Compared with children of the same age, does [name] have difficulty playing?	- Delete word or phrase	To simplify the item
Relationships	Identify children who have difficulty socializing with other children to an extent that it impacts their ability to participate in activities	Does [name] have difficulty getting along with children of his/her age?		-Delete item	To eliminate redundancy
		Does [name] have difficulty making and keeping friends?	Does (name) have difficulty making friends?	-Delete word or phrase	To simplify the item
		Compared with children of the same age, does [name] have difficulty playing with other children?		-Delete item	To eliminate redundancy
		Compared with children of the same age, does [name] have difficulty doing things with other children? (Include things that children usually do together.)		-Delete item	To eliminate redundancy

^*tested for children age 5–17 unless otherwise specified

^†not cognitively tested

^‡tested only in India and Jamaica

Wording Revisions

Several items contained language that was confusing or ambiguous. In these cases words were either revised or deleted. For example, as originally worded, the item intended to identify children with attention difficulties (Domain: Attention) was confusing and ambiguous to respondents. The original question asked respondents if their children had difficulty “concentrating on a task.” Some respondents were not sure what was meant by the term “task.” Other respondents noted that their children’s ability to complete tasks depended on factors such as the difficulty of the task or incentives provided for completion. A respondent in Belize said of her son, “It depends on the task. If it is of his interest, he has no problem.” To clear up confusion over the word “task,” the phrase “completing a task” was replaced with “concentrating on an activity” and the clause “that he/she enjoys doing” was added. Subsequent rounds of testing demonstrated that revisions were successful in clearing up these ambiguities.

The item intended to identify children with difficulties coping with change (Domain: Coping with Change) originally asked about accepting changes to “plans or routine.” While some respondents focused on changes to daily routines, many others focused on changes to anticipated plans. One respondent said, “You know kids… if you changed the plans —especially if he had his mind set on something and you changed it—of course [my son’s] going to be upset a little bit.” In order to simplify the item, reference to plans was removed in later rounds of testing.

Items intended to identify children with difficulties expressing and managing emotions (Domain: Emotions) were revised by adding the single word “very.” In earlier rounds of testing, respondents were not sure whether to answer based on the frequency of what they considered “normal” emotions or to only focus on extreme emotions. One respondent from Oman described his daughter’s “normal” anxiety:

It depends on the situation… I wouldn’t be concerned of her nervousness or anxiousness. Cause it might be like a roller coaster ride that she is scared to get on or something like that. It’s not like she is scared to enter a building or a room…

The addition of the qualifier “very” helped respondents focus on levels of anxiety or depression that impede functioning, which is what the items were intended to capture.

Revisions to Clarifying Phrases

Some items were revised by moving, deleting or adding phrases in order to clarify the scope of the question. For example, phrases such as “when wearing his/her glasses” or “when using his/her hearing aid” specify that respondents should answer based on difficulties with these aids. In the first round of testing, respondents often did not attend to these phrases which were tagged onto the ends of items. For example, in the first round of testing, the item intended to identify children with difficulty seeing (Domain: Seeing) read, “Does [name] have difficulty seeing [when wearing his/her glasses]?” Some respondents who had indicated that their children wear glasses, answered based on their children’s vision without glasses. Moving the “glasses” clause to the beginning of the item resolved this problem. Similarly, the “hearing aid” clause was moved to the beginning of the item on hearing (Domain: Hearing).

Initially the items intended to identify children with difficulties walking (Domain: Walking) did not specify whether respondents should evaluate their children’s walking with equipment or without. Therefore, respondents whose children used aids such as walkers, braces or gait trainers were not sure whether to answer based on their children’s ability to walk using their equipment or not. In later rounds of testing, a question on equipment or assistance for walking was added and the follow-up questions on walking added the clarifying phrases “with his/her equipment or assistance” and “without his/her equipment or assistance.”

Items in the communication domain originally asked respondents to consider their children’s ability to communicate “using his/her usual language.” Some respondents were not sure what was meant by “usual language” while others had various interpretations of the phrase. In particular, some respondents whose children were non-verbal interpreted the phrase to include communication through crying and gestures. The item was revised to include clarifying phrase “When [name] speaks…” Later rounds of testing demonstrated that this revision was successful in limiting the scope of respondents’ answers to verbal communication abilities.

In domains where functioning can change significantly across age groups, it may be appropriate for respondents to consider their children’s abilities relative to children of the same age. It would be appropriate for parents of 5 year olds to evaluate their children’s abilities compared to other 5 year olds but not compared to 16 year olds, for example. Therefore, initially, many items began with the phrase, “Compared to children of the same age…” For some domains such as seeing and hearing, respondents were easily able to make this comparison. However, in other domains such as controlling behavior, self-care, coping with change and attention it was difficult for respondents to make this comparison. These are domains where behavior occurs mostly in the home and is, therefore, less observable to other parents. In these domains, the comparison phrase was removed. Further testing confirmed that removal of the phrase eliminated confusion over the difficult comparison but did not hinder respondents’ ability to accurately respond.

Conceptual Revisions

Some items required revision in order to clarify their conceptual focus. The item on hearing (Domain: Hearing) was intended to focus on auditory hearing, but many respondents focused on listening or attention instead of or in addition to hearing. One respondent from the United States explained, “I call his name and he doesn’t respond. I don’t know if he’s ignoring me or if he really can’t hear. I mean, I know he can hear, but he doesn’t listen.” In order to clarify the conceptual ambiguity between hearing and listening, the item was revised to ask about “difficulty hearing sounds.” Although hearing is an extremely complex function that is related to auditory hearing, cognition, attention and motivation, results from the final round of testing demonstrated that the revised item was effective in focusing respondents’ responses away from interpretations related to listening and attention.

In the case of the communications question (Domain: Communication), initial rounds of testing indicated that respondents thought about communication in the household separately from communication outside the household and that children’s ability to communicate could vary greatly depending on the context. One respondent from Belize was unsure how to answer saying:

You would have to be around her to understand her more and even so you would have difficulty. But for family members my sisters, my mom, dad and aunties they talk to her, she respond, but it still hard for others to understand her. Like if a stranger is around they would not understand her at all.

To solve this problem, the communication item was split into two separate items, one asking about communication inside the household and the other asking about communication outside the household. This enabled respondents to indicate if their children’s ability to communicate differed between the two contexts.

The single item in the emotions domain was also improved by being split into two separate items. The original emotions question asked about both worry and sadness. Some respondents interpreted this as double-barreled. One respondent from India said, “I don’t really know. He does worry a lot, but on the other hand, he doesn’t really get sad, so I’m not sure.” Splitting this domain into two separate questions allowed respondents to focus on the relevant constructs separately.

Some items were deleted from the final instrument because they overlapped conceptually with other items and were, therefore, not necessary. Within the communication domain, an item on comprehension asked, “…does name have difficulty being understood by other people?” Initial testing showed that respondents interpreted this item in a variety of ways. For instance, some understood this as a question about vocabulary while others thought of it as a question about attention and still others answered based on hearing or language ability. Since this item was not understood consistently by respondents and because the basic elements of comprehension are captured elsewhere in the module in questions related to hearing and cognition, the item was deleted.

Use of examples

Examples can be used to clarify question intent by helping respondents generalize about a particular category. However, examples can also distract respondents and limit their responses. Therefore, if an item functions well without examples, it is best not to insert them needlessly. Testing was useful in identifying items where examples were needed. The walking questions (Domain: Walking) were first tested using the distances 100 and 500 yards (meters) without clarifying examples. Very few respondents could accurately describe the distances. Respondents in the United States estimated 500 yards to be anywhere from the length of one city block to several miles. In later rounds of testing a “country specific example” was inserted. In the United States, for the distance 100 yards, this was “about the length of one football field.” Subsequent testing demonstrated that the inclusion of the football field example increased respondents’ accuracy in estimating the distances.

Testing also supported the inclusion of examples in the revised hearing question (Domain: Hearing) which asked about hearing sounds “like voices or music” and in the self-care question (Domain: Self-Care) which used the examples “like feeding or dressing.”

Addressing Bias:

To find potential sources of bias, this study explored potential interpretive differences across respondent subgroups of respondents. Since the module on child functioning was intended as a cross-nationally comparable instrument to measure child disability, it was important to focus analysis on how cultural factors and disability status influenced the response process.

Cross-cultural factors

Examination of cross-cultural variations showed that question performance was, at times, rooted in cultural context. For example, respondents in India and Belize answered questions on communication from the context of a multi-lingual society while respondents in Oman answered walking questions from the perspective of a culture that, due to extreme heat and the availability of cars, does not depend on walking as a means of transportation. Revisions to questions often focused on reducing these differences. Asking communication questions that centered on communication both within and outside of the household directed respondents to focus more on their children’s communicative functioning and less on the linguistic variations of their regions. Using country specific examples in the walking questions helped respondents evaluate their children’s ability to walk in their usual environment.

Disability Status

Another area of subgroup analysis examined potential differences in how disability status influenced response patterns. Having a child with a disability might alter the way respondents interpret or respond to questions about child functioning. In fact, some differences were observed. Respondents whose children had disabilities were at times not sure whether to compare their children to other disabled children or to “typical” children. One respondent said, “I can’t compare [her child] to most other children his age because he’s not like them. His abilities are not at their level, but he does make progress in his own way.” Like this respondent, some parents of disabled children compared them to other disabled children while others compared their children to children without disabilities.

Additionally, parents of children without difficulties were often not sure what to consider a “normal” level of difficulty. One respondent indicated that her daughter had “some difficulty” hearing. She explained, “She does have some difficulty. If I say something to her, it’s like it goes in one ear and out the other. I know she heard me, but I don’t know if that’s just a normal thing- if she’s like other kids or I should worry that she’s just not listening all the time.” Parents of children without difficulties often answered “some difficulty” to signal minor behavioral concerns or to indicate that “everyone has SOME difficulty.” While some of the impact of disability status on the response process was ameliorated through item revision, there remained some lingering differences in the ways these questions function across subgroups.

Discussion

CI studies are designed to uncover areas where respondents’ cognitive paths to question response do not align with the original constructs behind the questions. That is, CI studies can reveal questions and answers that do not fully capture the measures that researchers want to collect. Further, because it is a qualitative method that relies on the collection of in-depth respondent narratives, CI can also reveal social and cultural elements that can introduce bias across respondent subgroups.

The present study supports the use of cross-national CI studies as an effective tool for developing cross-nationally comparable measures. Revisions were made iteratively based on the findings of each round of testing. Questions were revised when they were found to be vague, confusing or difficult to answer. Redundant questions were eliminated altogether. Where differences were found across subgroups of respondents, questions were revised to encourage alignment across these groups. Unsurprisingly, the language of instrument and interview was a significant factor in cross-cultural variation. Language-level revisions were made to the translated instruments based on the CI results. Throughout the process of testing and revision, question constructs were evaluated and re-evaluated for precision and utility.

While some error is inevitable in survey measures, many sources of error were reduced or eliminated through successive testing and revisions resulting in a set of questions that is conceptually sound. Similarly, it is not possible to eliminate all sources of bias. However, sources of bias identified during the development of the instrument, were further investigated during field testing (see Paper 3).

Previous cross-cultural CI projects have reported difficulties in interviewing non-English-speaking respondents. ^7,8,9 The present study demonstrated that, while some difficulties in collecting CI data may be tied to respondents’ cultural norms, it is possible to collect comparable data across cultural and linguistic groups. In this case, interviewer preparation and an open interview protocol were crucial in data quality. Interviewers underwent rigorous training, but equally importantly interviewers were local to the areas where the interviews were conducted. Thus, interviewers were skilled at eliciting respondents’ narratives and were also able to understand much of the cultural context surrounding narratives. A broad, semi-structured interview protocol allows interviewers to access respondents’ thought process in an indirect, non-intrusive fashion while allowing respondents the latitude to tell their own stories.

The limitations of this study were inherent to the CI methodology. CI requires time and specialized skills, and considerable resources were devoted to training interviewers in the countries where interviews were conducted. However, use of relatively inexperienced interviewers meant that not all interviewers were equally skilled at eliciting respondent narratives. Oversight by CI trainers and use of dedicated CI software (Q-Notes) at least partially mitigated these variations in data quality through transparency of the analytic process. Additionally, although the sample size was relatively large for a CI study, the use of a purposive sample does not support the statistical generalizability of findings.

Conclusion

CI methods contributed to the development of the UNICEF/WG module on child functioning. Findings from multiple rounds of cognitive testing were used to improve the content and structure of the module. Findings from CIs were used to reduce or eliminate sources of error and bias. These findings guided decisions about the inclusion, revision, framing and deletion of items. Findings from this evaluation serve to bolster the validity of items in the finalized module. Further, findings underscore the importance of careful translation and implementation of this and similar instruments intended for use in cross-cultural contexts.