The development and testing of a module on child functioning for identifying children with disabilities on surveys. III: Field testing

Abstract

Background

A module on child functioning developed by UNICEF and the Washington Group on Disability Statistics (WG) for use in censuses and surveys reflects current thinking around disability measurement and is intended to produce internationally comparable data. The Child Functioning Module (CFM) was developed in response to limitations of the Ten Question Screening Instrument (TQSI) for use in surveys and builds on the WG Short Set (WG-SS) of questions that was designed to capture disability in censuses, particularly among the adult population.

Objective

This paper documents the testing of the module and summarizes its results, including a description of prevalence levels across countries using different cut-offs, and comparisons with prevalence levels obtained using the TQSI and the WG-SS.

Methods

Field tests were conducted in Samoa as part of the 2014 Demographic and Health Survey and in Mexico as part of the 2015 National Survey of Boys, Girls and Women. The module was also implemented in Serbia as part of a dedicated survey conducted in the province of Vojvodina, in February 2016.

Results

Using the recommended cut-offcut-off, the percentage of children reported as having functional difficulty ranges from 1.1% in Serbia to 2% in Mexico among children aged 2–4 years, and from 3.2% in Samoa to 11.2% in Mexico among children aged 5–17 years. Across all three countries, the prevalence of functional difficulty was highest in the socio-emotional domains.

A comparison of the prevalence levels obtained using the WG-SS and the CFM shows that, except for the question on cognition/learning, the WG-SS and the CFM are relatively close for children aged 5–17 years for the domains that are included in both question sets, but the WG-SS excludes many children identified by the CFM in other domains. The comparison between the TQSI and the CFM shows that, while the prevalence estimates are similar for seeing and hearing, significant differences affect other domains, particularly cognition/learning and communication.

Conclusions

The CFM addresses a full range of functional domains that are important for child development. The module represents an improvement on the TQSI in that it allows for scaled responses to determine the degree of difficulty, and so can separate out many potential false positives. The module is also preferred over the WG-SS for collecting data on children, first, because most of the questions in the WG-SS are not suitable for children under the age of 5 years, and second, because the WG-SS leaves out important functional domains for children aged 5–17 years, namely those related to developmental disabilities and behavioural issues.

This article summarizes the results of field-testing the UNICEF/Washington Group (WG) Child Functioning Module (CFM) in Samoa, Serbia and Mexico between 2014 and 2016. The module was created in collaboration with experts in child development, statisticians from national statistical offices and representatives from disabled people’s organisations and international agencies to identify children with disabilities in household surveys (see paper I). Prior to the field tests, cognitive testing of the module was conducted in several countries, including Belize, India, Jamaica, Montenegro, Oman and the United States (see paper II).

In Samoa, the Bureau of Statistics conducted the field test in collaboration with the Ministry of Health as part of the 2014 Demographic and Health Survey.¹ The field test in Serbia was conducted by the Statistical Office of the Republic of Serbia as part of a dedicated survey administered in the province of Vojvodina, in February 2016. WG and UNICEF provided technical and financial assistance in both countries. In Mexico, the module was implemented as part of the 2015 National Survey of Boys, Girls and Women (ENIM)² conducted by the National Institute of Public Health, under the fifth round of the UNICEF-supported Multiple Indicator Cluster Survey (MICS) programme.

The primary objective of these field tests was to collect data that could inform the adoption of the recommended cut-offs for identifying children with a disability. The tests also provided an opportunity to gather information that could improve implementation of the CFM, with a focus on:

• identifying specific needs for training interviewers;

• testing the flow of the questions and the skip patterns;

• gauging the amount of time needed to complete the survey; and

• providing a rough estimate of the positive response rate to assist in calculating the required sample size.

In Serbia, a secondary objective was twofold: to compare the CFM results with results from the Ten Question Screening Instrument (TQSI) in the case of children aged 2–4 years; and to compare the CFM results with results from the Washington Group Short Set (WG-SS) of questions in regard to children aged 5–17 years.

Methods

In Samoa, Serbia and Mexico, the CFM was administered to the mother or to the primary caregiver (if the mother was deceased or did not reside in the household) of all children aged 2–17 years; two separate sets of questions were used, tailored to the child’s age (2–4 years or 5–17 years). The same version of the module was used in all three countries, with some exceptions. For example, the question related to fine motor skills was not included in the Samoa survey because it was added to the standard CFM after the field test was completed. In addition, the response options used in Samoa for the question related to controlling behaviour among children aged 5–17 years were “a lot of difficulty” instead of “more difficulty,” and “cannot do at all” instead of “a lot more difficulty.” The questionnaire was translated into the national languages (Samoan, Serbian and Spanish) and then back translated. Differences in translation were resolved through focus group discussions between the translation teams and the survey technical leads.

In Serbia, the TQSI and the WG-SS modules were also used for the 2 to 4 and 5 to 17 age groups, respectively, as points of comparison for the Child Functioning Module. All three surveys also collected data on personal and household characteristics, such as sex and age of the children, mother’s education and household wealth.

In addition to the standard questions on child functioning, the versions of the module used in the three countries included a series of probing questions that were introduced to determine possible misinterpretations by parents of the concepts embedded in the questions, and consequently, out-of-scope responses. These probes were developed for exclusive use in the field tests and are not part of the module. In particular, the probes aimed at providing insight into the nature and severity of the functional difficulties reported by respondents. The probes were used for questions on walking for both age groups (2–4 and 5 to 17) and for questions on self-care, remembering, controlling behaviours and accepting changes for children aged 5–17 years. The inclusion of the probes was driven by the need to gauge levels of false positive cases resulting from questions that elicited a higher than expected proportion of out-of-scope responses during cognitive testing, particularly among parents of younger children (see paper II). Results from cognitive testing indicate that children with disabilities were systematically reported as having functional difficulties by their parents, and therefore probes were not used to detect false negatives.

In Samoa, the sample was drawn from the master sample frame for the 2011 Population and Housing Census, covering 16% of households in rural areas and 17% in urban areas and appropriate for generating indicators in four regions (Apia Urban Area, North West Upolu, the rest of Upolu and Savaii). A representative sample was selected in two stages – first from clusters in the master sample, and then from a complete listing of households from the 2011 census frame. During the first stage of the sample selection, 458 primary sampling units were identified (132 in urban areas and 326 in rural areas). In the second stage, a fixed number of 7 households per cluster in urban areas and 10 households per cluster in rural areas was selected using equal probability systematic selection. The result was a final sample of 4171 households with 9565 children aged 2–17 years (2139 children aged 2 to 4; 7426 children aged 5 to 17), yielding a response rate of 82% for children.

In Mexico, the ENIM 2015 was a multistage, cluster and stratified national survey designed to produce estimates on 136 indicators of well-being for women and children, with the possibility to stratify by rural or urban areas and by five regions within Mexico (Northwest, Northeast, Central, Mexico City-State of Mexico and South). Sampling followed a probabilistic, multistage, cluster and stratified design.³ Clusters corresponded to the basic geo-statistical areas defined by the National Institute of Statistics and Geography and constituted the primary sample units (PSUs). In urban areas, clusters were made up of blocks; in rural areas, clusters were made up of places or municipalities, with “rural” areas defined as those having a population of less than 2500. The ENIM 2015 sample frame was built from previously available censuses with geo-statistical data, which were updated through a cartographic listing exercise as part of the preliminary survey activities. For the sample design, five regions composed of neighbouring states were defined, each region corresponding to an equivalent population size. The sample was stratified in each of these five regions as well as in urban and rural areas. Thus defined, the final sample included 11,825 households with an oversample of households with children younger than 5 years (N = 8216 children); consequently, this resulted in a higher proportion of women of reproductive age (N = 12,937) and of 5–17 year old children (N = 11,812), with a response rate of 98% for children. Seven originally selected PSUs were replaced due to insecurity affecting the country during the time of household listing and data collection. In total, 17 PSUs were added during the third stage of selection. Standardized sample weights were calculated to account for non-response and specific selection probabilities.

In Serbia, the field test was carried out only in the province of Vojvodina and used two different samples. The Serbian samples were designed to provide estimates for a large number of indicators on the situation of children aged 2–17 years for Vojvodina and for the “urban” and “other” domains.¹ The urban and other domains within seven areas were identified as the main sampling strata; within each stratum, a specified number of census enumeration areas was selected systematically, with probability proportional to size. The samples were selected in two stages. A random sample of enumeration areas (clusters of households) was selected with probability proportional to size at the first stage, in which the measure of “size” was based on the number of households in the 2011 census frame. At the second stage, two samples of households (one for each questionnaire) were selected in the same enumeration areas. The overall number of households selected per cluster was determined as 26 households, or 13 households for each questionnaire. This decision was based on a number of considerations, including the budget available and the time frame needed per team to complete one cluster. After the selection of enumeration areas, households were divided into those with and those without children aged 0–13 years, and a separate systematic sample of households was selected for each group and questionnaire. At the Vojvodina level, a total of 3737 households was selected (2465 with children and 1272 without children). Since the Population Census from 2011 was used for the sample frame and the fieldwork was conducted in 2016, households without children were included to capture children aged 2–4 years who might have been born during this period in those households that did not have children at the time of the census. Separate samples of households for each questionnaire were selected from each second-stage stratum, using a higher sampling rate for households with children aged 0–13 years. Of the total households. 2915 mothers or caretakers were interviewed, and questionnaires were completed for 2913 children aged 2–17 years (374 children aged 2–4 years and 2539 children aged 5–17 years) with a response rate of 85% for children.

In Samoa, survey field personnel received 30 days of training, including question-by-question explanations of the entire survey instrument, a review of the interviewer’s manual, and practice interviews and tests. Sector specialists were invited to deliver lectures on each of the main topics covered in the surveys. This included a dedicated presentation of the disability questionnaire, which was followed by practice interviews and a focus group discussion with interviewers on the individual questions. Fieldwork was carried out between July and August 2014 by four teams, each composed of one supervisor, two field editors, between six and nine female interviewers, and five to seven male interviewers. Initial data editing occurred in the field, then double data entry was conducted using CSPro, with 100% verification. Concurrent processing of the data was used to alert teams in the field to possible problems detected during data entry. Data entry was completed in November 2014 for final tabulation using SAS in January 2015.

In Mexico, data collection was carried out from September to December 2015 by 12 teams of three interviewers, a supervisor and a nurse in charge of the anthropometric measurement. Interviews were conducted using a computer-assisted personal interview instrument, administered with tablet computers. Survey field personnel received 18 days of training, including discussion of ethical protocols, question-by-question explanations of the survey instrument and practice interviews. Tablet computers were used to collect the data. As in Samoa, the interviewers were trained on the entire survey questionnaire and attended a dedicated session on the disability questions, which was followed by a focus group discussion and practice interviews. All the applications for data collection and management were programmed using CSPro, version 5.

In Serbia, six teams composed of four interviewers, a supervisor and a controller per team were trained over five days prior to commencing fieldwork. Training consisted of lectures on interviewing techniques, familiarization with the content of the survey and opportunities to practice interviewing. Given that the survey in Serbia had a focus on disability and was designed with the specific objective of testing the new CFM, the interviewers received more extensive training on the module compared to interviewers in Samoa and Mexico. This included more time for practice interviews and focus group discussions on the individual questions. Feedback on the implementation of the module, specifically concerning parents’ responses to sensitive questions and challenges in administering interviews, was solicited from the interviewers and field editors during and after interviews. The teams carried out the interviews from January to February 2016. Eight operators fed data into eight computers using the CSPro software, version 6.0. To ensure data quality control, all responses were entered twice and internal consistency was verified.

In consistence with the objectives of the field tests, the article uses the data collected using the CFM to explore different severity cut-offs for determining disability prevalence, using the following criteria:

• consistency in prevalence levels;

• conformity to expected patterns across domains and within sociodemographic groups based on past studies;⁴

• prevalence of false positive cases based on caregivers’ responses to probes; and

• analysis of interviewers’ feedback on the implementation of the module to detect difficulties with the respondents’ understanding of the questions and their ability and willingness to answer them under field conditions.

The presentation of the results from Mexico, Serbia and Samoa is done jointly to illustrate how the module performed across countries and whether the same patterns hold true in the three settings. The statistical significance of variation in prevalence levels across sociodemographic groups was assessed on the basis of results from the Pearson’s chi-square test (at the p < 0.01 level) and confidence intervals (whether or not they overlapped).

Finally, the paper compares results from the CFM, the TQSI and the WG-SS to illustrate differences in prevalence levels due to different scope and design of the three instruments.

Data were processed at the country level using the statistical software packages noted above. The analysis presented in this paper was conducted using Stata, version 14.1.

Results

Establishing severity cut-offs to determine disability prevalence

Data resulting from the CFM in the three countries are used to make comparisons about the sensitivity of disability prevalence estimates by applying different cut-offs or thresholds of severity and identifying how those relate to the characteristics of children with disabilities. As the module contains a number of questions, all with scaled responses, there are potentially many ways to establish a cut-off for creating a dichotomous category of disability/no disability, in order to establish an overall disability prevalence. Table 1 presents three different approaches to establishing that cut-off, corresponding to several different severity levels.

Table 1.

Definitions of different cut-offs for identifying children with functional difficulty among children aged 2–4 years and 5–17 years, using the Child Functioning Module.

Cut-offs	Children aged 2–4 years	Children aged 5–17 years
At least one domain coded as follows:
Cut-off 1 (most inclusive)	• “Some difficulty”, “A lot of difficulty” or “Cannot do at all”	• “Some difficulty”, “A lot of difficulty” or “Cannot do at all”
	• “More” or “A lot more” for controlling behaviour	• “Weekly” or “Daily” for anxiety and depression
		• “More” or “A lot more” for controlling behaviour
Cut-off 2	• “A lot of difficulty” or “Cannot do at all”	• “A lot of difficulty” or “Cannot do at all”
	• “More” or “A lot more” for controlling behaviour	• “Daily” for anxiety and depression
		• “More” or “A lot more” for controlling behaviour
Cut-off 2A	• “A lot of difficulty” or “Cannot do at all”	• “A lot of difficulty” or “Cannot do at all”
	• “A lot more” for controlling behaviour	• “Daily” for anxiety and depression
		• “A lot more” for controlling behaviour
Cut-off 3 (most restrictive)	• “Cannot do at all”	• “Cannot do at all”
	• “A lot more” for controlling behaviour	• “Daily” for anxiety and depression
		• “A lot more” for controlling behaviour

Cut-off 1 includes all children who have any degree of difficulty in at least one domain, and thus represents the lowest threshold for identifying children with functional difficulties. Cut-off 3 includes only children with the highest level of difficulty in at least one domain. Using Cut-off 2, the intermediate category, children aged 2–17 years are considered to have functional difficulty if they are reported to have “more” or “a lot more” difficulty controlling their behaviour or at least “a lot” of difficulty in one of the other domains. In addition to these criteria, children aged 5–17 years who have “daily” episodes of anxiety or depression would also be considered to have functional difficulty. Cut-off 2A applies the same thresholds as Cut-off 2 for children, however, it uses a higher threshold for the behavioural domain, so children reported as having only “a lot more” difficulty in this area are considered to have functional difficulty.

Other cut-offs were examined for children aged 5–17 years, specifically to determine how responses to questions related to anxiety, depression, and behaviour should be used in determining prevalence. For anxiety and depression, a more stringent cut-off was chosen (“daily” over “weekly”) in order to restrict the focus on the more severe cases and to account for any underlying environmental or developmental factors (e.g., the onset of puberty) that would cause stress to the general population.

Analysis of disability prevalence across cut-offs among children aged 2–4 years

Table 2 shows how prevalence levels decrease in each country as the more stringent cut-offs are employed. The most inclusive measure of disability, Cut-off 1, which includes all children with any level of difficulty, yields prevalence estimates that are much higher than the other thresholds. The largest differences across countries are observed using Cut-off 1, which is aligned with the findings of the cognitive testing (see paper II), namely that there might be more variance (within and across countries) regarding how respondents interpret the “some difficulty” response category. Using Cut-offs 2 and 2A, which are analogous to the Washington Group recommended cut-off for identifying people with disabilities using the WG-SS (having at least “a lot” of difficulty in at least one functional domain), the overall prevalence in Samoa and Serbia is similar, but it remains higher in Mexico.⁵

Table 2.

Percentage of children aged 2–4 years reported as having functional difficulty based on different cut-offs, by child’s background characteristics.

Total	Mexico				Samoa				Serbia
	Cut-off 1	Cut-off 2	Cut-off 2A	Cut-off 3	Cut-off 1	Cut-off 2	Cut-off 2A	Cut-off 3	Cut-off 1	Cut-off 2	Cut-off 2A	Cut-off 3
	27.5	5.4	2	0.4	15.5	2.8	1.4	0.8	11.5	3.8	1.1	0.0
Sex
Boys	29.4	7.4	2.5	0.5	17	3.1	1.8	1	16.3	5.2	0.5	0.0
Girls	25.7	3.7	1.5	0.3	13.8	2.5	0.8	0.3	6.4	2.4	1.2	0.0
Mother’s Education
None	36.3	17.1	8.3	0.0	NA	NA	NA	NA	*	*	*	*
Primary	27.4	7.3	2.9	0.7	NA	NA	NA	NA	(8.6)	(4.2)	(2.5)	(0.0)
Secondary	28.2	4.9	1.7	0.4	NA	NA	NA	NA	9.1	4.8	0.6	0.0
Tertiary	22.8	3.7	1.0	0.0	NA	NA	NA	NA	12.3	1.6	0.0	0.0
Missing	*	*	*	*	NA	NA	NA	NA	–	–	–	–
Residence
Urban	27.2	5.2	1.9	0.4	16.3	3.9	1.9	1.1	8.5	1.9	0.8	0.0
Rural	28.2	6.3	2.2	0.3	15.4	2.6	1.2	0.6	15.5	6.4	1.5	0.0
Wealth
Poorest	30.4	9.1	3.0	0.4	18.7	3.8	1.2	0.8	(10.9)	(6.4)	(3.9)	(0.0)
Second	25.9	5.9	2.0	0.7	14.8	3.4	2.3	1.4	(18.9)	(1.9)	(0.0)	(0.0)
Middle	33	4.1	2.2	0.4	15.7	2.1	1.4	0.5	(3.7)	(3.7)	(1.0)	(0.0)
Fourth	24.1	4.3	1.2	0.2	13.5	2.5	1.2	0.3	(6.2)	(0.0)	(0.0)	(0.0)
Richest	21.1	1.9	0.7	0.0	14	2.0	0.6	0.6	(20.6)	(5.9)	(0.0)	(0.0)

Note:

^*indicates values based on fewer than 25 cases. Values in parentheses are based on fewer than 50 unweighted cases. Complete information on mothers’ level of education is unavailable for children in Samoa.

Prevalence varies to some extent across populations based on the cut-off used to identify children as having functional difficulties, although the effect was minimal and the patterns remain consistent. Prevalence appears to be higher among boys than girls in all countries, with the exception of Serbia when using Cut-off 2A. In Mexico, when using Cut-off 2, the differences in prevalence levels between boys and girls were significant based on results from the Pearson chi-square test (p < 0.01) with confidence intervals not overlapping (not shown). Prevalence in Mexico and Serbia also appears higher among children whose mothers have none or primary-level education compared to those whose mothers completed secondary schooling or higher. For Mexico, the differences were observed at the p < 0.01 level based on the Pearson chi-square test, with confidence intervals not overlapping when using Cut-offs 2 and 2A.

Although differences in prevalence by place of residence were not observed at the p < 0.01 level based on the Pearson chi-square test, higher levels were noted among children in rural areas compared to those in urban settings for all cut-offs except Cut-off 3, in both Mexico and Serbia. The opposite trend is observed in Samoa, where prevalence is consistently higher among children aged 2–4 years in urban areas. Some variation is also observed in patterns of prevalence for household wealth quintile when using different cut-offs, specifically for the second and middle quintiles. However, differences were only significant at the p < 0.01 level based on the Pearson chi-square test in Mexico when using Cut-off 2, with confidence intervals for the poorest and second quintiles not overlapping with those for the richest quintile.

As shown in Table 3, across all three countries, the percentage of children with “more” or “a lot more” difficulty controlling their behaviour is higher than the percentage of children with only “a lot more” difficulty controlling behaviour or with difficulty in any of the other domains. When using Cut-off 2A, which employs the more stringent cut-off for the question on controlling behaviour, the percentage of children aged 2–4 years with functional difficulty was highest for being understood in Mexico and Serbia and remained highest for the behavioural domain in Samoa (0.5%). Results show that, in all three countries, less than 1% of children aged 2–4 years had functional difficulties in more than one domain.

Table 3.

Percentage of children aged 2–4 years with severe functional difficulty, by domain, based on Cut-offs 2 and 2A

Functional Domain	Mexico	Samoa	Serbia
Seeing (“A lot of difficulty” or “Cannot do at all”)	0.2	0.1	0.5
Hearing (“A lot of difficulty” or “Cannot do at all”)	0.2	0.1	0.0
Walking (“A lot of difficulty” or “Cannot do at all”)	0.3	0.4	0.0
Fine motor (“A lot of difficulty” or “Cannot do at all)	0.1	–	0.0
Understanding (“A lot of difficulty” or “Cannot do at all”)	0.3	0.4	0.0
Being understood (“A lot of difficulty” or “Cannot do at all”)	1.0	0.4	0.6
Learning (“A lot of difficulty” or “Cannot do at all”)	0.3	0.4	0.0
Playing (“A lot of difficulty” or “Cannot do at all”)	0.2	0.3	0.0
Controlling behaviour (“More” or “A lot more”)	4.0	2.1	3.2
Controlling behaviour (“A lot more”)	0.2	0.5	0.0
Functional difficulty in at least one domain (Cut-off 2)	5.4	2.8	3.8
Functional difficulty in more than one domain (Cut-off 2)	0.6	0.6	0.5
Functional difficulty in at least one domain (Cut-off 2A)	2.0	1.4	1.1
Functional difficulty in more than one domain (Cut-off 2A)	0.4	0.5	0.0
Total number of children	5153	2135	219

Note: The question related to fine motor skills was not included in the Samoa survey.

The figures in Table 4 indicate that, among children who were identified as having functional difficulty based on Cut-off 2, 60% in Samoa and 67% in Mexico had difficulty controlling their behaviour but had no difficulty in any other domains. When applying Cut-off 2A, the percentage of cases attributed only to difficulty in the behavioural domain decreases to 31% in Samoa and 10% in Mexico. When using Cut-off 2A in Mexico, 35% of children aged 2–4 years who were identified as having functional difficulty had difficulty being understood but no difficulty in any other areas.

Table 4.

Cut-off Percentage of children with difficulty in only one domain and in more than one domain among children aged 2–4 years with functional difficulty based on Cut-offs 2 and 2A

Functional Domain	Mexico	Samoa
	Cut-off 2	Cut-off 2A	Cut-off 2	Cut-off 2A
Only Seeing	2.3	6.4	3.0	(10.3)
Only Hearing	2.2	6.1	1.6	(3.4)
Only Walking	2.2	7.3	0.0	(3.4)
Only Fine Motor	0.8	2.4	–	–
Only Understanding	1.1	3.8	3.0	(6.9)
Only Being Understood	9.5	35.0	3.0	(6.9)
Only Learning	2.4	6.7	1.6	(3.4)
Only Playing	1.0	2.8	1.6	(3.4)
Only Controlling Behaviour	67.7	10.4	60.0	(31.0)
More than one domain	10.8	19.1	26.2	(31.3)
Total number of children with functional difficulty	280	101	60	29

Note: The question related to fine motor skills was not included in the Samoa survey. The number of cases in Serbia was <25, so values are not shown. Values in parentheses are based on fewer than 50 unweighted cases.

Analysis of disability prevalence across cut-offs among children aged 5–17 years

As illustrated in Table 5, prevalence of functional difficulty among children aged 5–17 years varied across countries based on the use of different cut-offs, most noticeably for Cut-off 1. One quarter of children in Serbia (25.2%) and almost half in Mexico (46.3%) were identified as having functional difficulty based on Cut-off 1. As with trends observed among the 2 to 4 age group, prevalence estimates among children aged 5–17 years were similar in Samoa and Serbia but noticeably higher in Mexico. With the exception of Serbia, prevalence among children aged 5–17 years was higher among boys than girls; in Mexico, when employing Cut-offs 2 and 2A, the differences observed by sex were significant at the p < 0.01 level based on the Pearson chi-square test with confidence intervals not overlapping.

Table 5.

Percentage of children aged 5–17 years with severe functional difficulty, by child’s characteristics and cCut-off.

Total	Mexico				Samoa				Serbia
	Cut-off 1	Cut-off 2	Cut-off 2A	Cut-off 3	Cut-off 1	Cut-off 2	Cut-off 2A	Cut-off 3	Cut-off 1	Cut-off 2	Cut-off 2A	Cut-off 3
	46.3	14.1	11.2	6.5	9.3	3.3	3.2	2.4	25.2	4.5	3.6	2.2
Sex
Boys	47.0	16.3	12.7	7.2	9.8	3.6	3.4	2.5	23.8	4.4	3.6	2.1
Girls	45.6	11.9	9.8	5.7	8.7	3	3.0	2.3	26.6	4.7	3.6	2.3
Mothers’ education
None	41.8	13.6	11.9	7.2	NA	NA	NA	NA	42.3	17.6	13.6	6.3
Primary	48	16	12.8	6.5	NA	NA	NA	NA	38.1	6.3	5.3	3.4
Secondary	46.6	13.4	10.8	6.5	NA	NA	NA	NA	19.1	3.3	2.6	2
Tertiary	42.4	13.2	9.3	6.2	NA	NA	NA	NA	26.8	3.4	2.7	0.8
Missing	37.4	10.8	10.8	3.6	NA	NA	NA	NA	–	–	–	–
Residence
Urban	47.3	14.3	11.6	6.9	12.3	2.5	2.2	1.5	26.4	4.2	3.7	2
Rural	43.5	13.7	10.2	5.2	8.6	3.5	3.4	2.6	23.3	5	3.5	2.5
Wealth
Poorest	45.5	13.5	10.9	5.8	10.8	4.1	4.1	2.7	32.5	8.1	7.1	4.2
Second	46.9	15.8	12.5	7.1	8.1	3.0	2.9	2.1	32	5.5	4.6	3
Middle	47.8	16.5	13.4	8.4	8.0	2.5	2.5	1.9	25.1	1.5	1.0	0.6
Fourth	49.9	13.4	10.6	5.1	10.6	3.4	3.1	2.5	19.1	3.6	2.9	0.2
Richest	41.3	10.6	8.0	5.6	8.7	3.4	3.3	2.7	16.3	3.3	1.8	0.8

Note: Complete information on mothers’ level of education is unavailable for children in Samoa.

Consistent with the findings observed for the younger age group, place of residence did not appear to be strongly associated with prevalence of functional difficulty among children aged 5–17 years. In Mexico, although prevalence appeared consistently higher among children living in urban areas compared to their rural peers, the differences observed were not significant at the p < 0.01 level based on the Pearson chi-square test, and confidence intervals overlapped.

In Serbia and Samoa, prevalence remained highest among children from the poorest households regardless of the specific cut-off applied. In Serbia, the differences observed across the wealth quintiles were significant at the p < 0.01 level based on the Pearson chi-square test for all cut-offs. Prevalence in Mexico was highest among children from the middle household wealth quintile using cut-offs 2, 2A and 3, although the variation was not statistically significant.

The inverse relationship between prevalence and mother’s educational attainment observed for children aged 2 to 4 in Mexico was less strong among the older age group. However, in Serbia, prevalence was highest among children aged 5 to 17 whose mothers had no education for all cut-offs (p < 0.01 based on the Pearson’s chi-square test).

As shown in Table 6, the prevalence of functional difficulty across countries was highest in the socio-emotional domains, specifically anxiety and controlling behaviour. The percentage of children who were reported as experiencing “daily” episodes of anxiety ranged from 2% in Samoa and Serbia to almost 6% in Mexico.

Table 6.

Percentage of children aged 5–17 years with functional difficulty based on Cut-offs 2 and 2A, by functional domain.

Functional Domain	Mexico	Samoa	Serbia
Seeing	0.8	0.1	0.5
Hearing	0.2	0.4	0.0
Walking	0.9	0.2	0.3
Self-care	0.5	0.2	0.2
Being understood inside household	0.6	0.3	0.3
Being understood outside household	1.0	0.3	0.4
Learning	1.9	0.5	0.9
Remembering	1.3	0.4	0.7
Focusing	1.3	0.3	0.0
Accepting change	1.8	0.2	0.4
Making friends	1.8	0.3	0.8
Anxiety	5.5	1.8	1.7
Depression	1.5	1.4	0.4
Controlling Behaviour (More or A lot more)	5.2	0.3*	1.8
Controlling Behaviour (A lot more)	0.8	0.0*	0.5
Functional difficulty in at least one domain (Cut-off 2)	14.1	3.3	4.5
Functional difficulty in more than one domain (Cut-off 2)	4.6	1.7	1.4
Functional difficulty in at least one domain (Cut-off 2A)	11.2	3.2	3.6
Functional difficulty in more than one domain (Cut-off 2A)	3.9	1.7	1.3
Total number of children	11607	7426	1250

^*In Samoa, the response options for controlling behaviour were “a lot of difficulty” instead of “more difficulty” and “cannot do at all” instead of “a lot more difficulty.”

The figures in Table 7 indicate that, in all three countries, between 19 and 28% of the children identified as having functional difficulty based on both Cut-off 2 and Cut-off 2A were reported to experience daily anxiety but were said to have no difficulty in any of the other domains. When using Cut-off 2, the percentage of children reported as having “more” or “a lot more” difficulty controlling their behaviour was 5% in Mexico, 2% in Serbia, and less than 1% in Samoa. It is possible that the use of different response options for this question in Samoa resulted in more accurate reporting of difficulty in this area, thus explaining the deviation from the high prevalence of difficulty in this domain that was observed in Mexico and Serbia. Based on Cut-off 2, 20% of children in Mexico and 26% in Serbia who were identified as having functional difficulty were reported to have “more” or “a lot more” difficulty controlling their behaviour, but no difficulty in any of the other functional domains. Conversely, when applying Cut-off 2A, less than 2% of children identified as having functional difficulty in Mexico and 7% in Serbia were reported to have difficulty controlling their behaviour but no difficulty in any other areas.

Table 7.

Percentage of children with difficulty in only one domain and in more than one domain among children aged 5–17 years with functional difficulty based on Cut-offs 2 and 2A

Functional Domain	Mexico		Samoa		Serbia
	Cut-off 2	Cut-off 2A	Cut-off 2	Cut-off 2A	Cut-off 2	Cut-off 2A
Only Seeing	4.1	5.3	2.4	2.5	7.3	9.2
Only Hearing	0.8	1.0	6.5	6.8	0.0	0.0
Only Walking	2.2	3.1	0.4	0.4	3.2	4.0
Only Self-Care	0.6	0.7	0.0	0.0	0.0	0.0
Only Being Understood inside household	0.0	0.0	0.0	0.0	0.0	0.0
Only Being Understood outside household	0.4	0.6	1.2	1.3	1.2	1.5
Only Learning	2.6	4.2	3.3	3.4	1.2	1.4
Only Remembering	1.1	1.4	0.0	0.0	0.0	0.0
Only Focusing	2.8	4.3	2.4	2.5	0.0	0.0
Only Accepting Change	3.6	5.4	0.8	0.8	2.9	3.6
Only Making Friends	4.7	6.4	2.4	2.5	5.3	9.8
Only Anxiety	20.5	27.9	20	20.7	19.3	24.1
Only Depression	2.4	3.2	4.5	5.9	2.2	2.7
Only Controlling Behaviour	21.9	1.8	3.3	0.0	25.8	7.0
Difficulty in more than one domain	32.3	34.7	52.8	53.2	31.6	36.6
Number of children with severe functional difficulty	1638	1304	245	237	56	45

Summary of recommended cut-offs

For children aged 2–4 years, these results support the adoption of the recommended cut-off for identifying children with a disability as inclusive of children with “a lot more” difficulty controlling behaviours compared to their peers and at least “a lot” of difficulty in the other functional domains, as designated by Cut-off 2A. The use of such cut-offs yields estimates that appear to be more consistent within and across countries. Also, the adoption of such cut-offs will significantly reduce possible false positives, as opposed to a broader definition that would include children with any difficulty (see Annex A). The use of a more restrictive threshold for the behavioural domain in Cut-off 2A compared to Cut-off 2 is reaffirmed by results of the cognitive testing, which indicated that caregivers could potentially confuse functional difficulty in this area with behavioural outbursts that can be typical of young children (see paper II).

For children aged 5–17 years, the data support the adoption of the recommended cut-off for identifying children with a disability as inclusive of children with daily episodes of anxiety and depression and at least “a lot” of difficulty in the other functional domains, as designated by Cut-off 2. While the question on controlling behaviour seems to produce higher prevalence levels of false positives, it is worth mentioning that this domain of functioning involves more variance in expectations of what a child should be able to do at a certain age, which could vary depending on several factors, including culture and even a difference in age. The analysis of the Samoa data suggest that the use of different response categories for the question on behaviour – namely “a lot of difficulty” instead of “more difficulty” and “cannot do at all” instead of “a lot more difficulty” – might have generated lower prevalence levels and fewer false positives. For this reason, it was decided that moving forward the CFM would use the same response categories for the controlling behaviour question that were used in Samoa rather than the categories used in Mexico or Serbia.

Comparison of results between the CFM and TQSI and WG-SS

The Serbia field test data allow for comparison between the CFM and the TQSI and WG-SS. Parents of children aged 2–4 years in the comparison group received the TQSI, while parents of older children received the WG-SS. The TQSI was designed as a screen to identify all children who may have a disability. As such, without the accompanying second stage assessment, which is quite complex to administer, it is expected to generate high prevalence estimates of disability, including high proportions of false positive cases.⁶ Another factor that can lead to high prevalence is the use of the “Yes/No” response in the TQSI, which makes it impossible to identify children with mild and/or moderate levels of difficulty. If both stages of the TQSI methodology are used, these false positives can be eliminated, but that is not possible if only the first stage is used.

In fact, the TQSI in Serbia yielded a disability prevalence of 16%, compared to the 3.8% from using Cut-off 2 for children aged 2–4 years.² The WG-SS questions, on the other hand, were thought to be under-identifying children with disabilities because of the exclusion of developmental and behavioural domains that are important for child functioning but not as relevant for adults. This expectation was also met, as the overall disability prevalence for children aged 5–17 years using the WG-SS was 1.3%³ compared with 4.5% using the CFM with Cut-off 2.

Table 8 shows a comparison of the prevalence of disability by type using the CFM, TQSI and WG-SS for functional domains addressed by any of the questionnaires. Except for the question on cognition/learning, the CFM and the WG-SS are relatively close for children aged 5–17 years for the domains that are included in both question sets, but the WG-SS excludes many children identified by the CFM in other domains. The comparison between the CFM and the TQSI shows that while the prevalence estimates are similar for seeing and hearing, significant differences affect other domains, particularly cognition/learning and communication. This might be a result of the difficulty respondents had in interpreting the communication question in the TQSI. In sum, a direct comparison of the CFM to both the TQSI and the WG-SS indicates that the TQSI over-identified children with disabilities while the WG-SS under-identified the functional limitations of children, given its focus on only a subset of functional domains.

Table 8.

Comparison of disability prevalence levels by type using the CFM, TQSI and WG-SS for functional domains common to the three questionnaires.

Functional Domain	Children aged 2 to 4		Children aged 5 to 17
	CFM (Cut-off 2A)	TQSI	CFM (Cut-off 2)	WG-SS
Seeing	0.5	0.6	0.5	0.3
Hearing	0.0	0.0	0.0	0.2
Walking	0.0	0.8	0.3	0.2
Cognition/Learning	0.0	9.1	0.9	0.3
Self-Care	NA	NA	0.2	0.3
Communication	0.6	6.5	0.4	0.5
Total Prevalence (using all the questions that are part of the modules)	1.1	16.0	4.5	1.3

Limitations

Differences in survey administration, particularly regarding training protocols and survey questions, might have influenced the prevalence outcomes. Variation in prevalence estimations across the national contexts could also be due to different cultural understandings or to different levels of accommodation within each society.

Conclusions

The Child Functioning Module developed by UNICEF and the Washington Group was designed to be consistent with the notion of disability underlying the and so can separate out many potential false positives (CRPD) and the International Classification of Functioning, Disability and Health (ICF). In line with the psychosocial model of disability embedded in the ICF and the CRPD, the CFM aims at capturing activity limitations that, in an unaccommodating environment, would place a child at higher risk of participation restrictions than children without similar limitations. Impairments, diagnoses or conditions, on the other hand, are emblematic of the medical model of disability, the identification of which would require medical assessments. Such assessments in children necessitate resources that remain largely unavailable in most low- and middle-income countries and would have significant implications on data collection, given the need to conduct two stages of data collection. Instead, difficulties in functioning can be reported by primary caregivers, provided that they are asked clear questions that leave no room for equivocal interpretation of the child’s ability or inability to perform the selected function.

Implementation of the CFM in Serbia, Samoa and Mexico provides evidence in support of the use of the module in surveys as a reliable and easy-to-administer method for identifying children with functional difficulties. The module represents an improvement on the TQSI in that it allows for scaled responses to determine the degree of difficulty, and so can separate out many potential false positives while allowing for the identification of children who may be having problems and are potentially in need of other services. By reporting high levels of sensitivity and specificity of the module based on parental responses compared against medical assessments, a number of studies that employed the CFM as a screening tool in a two-stage data collection process have confirmed such strengths.^9,10

The CFM also addresses a broader range of functional domains that are important for child development. As such, the module is preferred over the WG-SS for use in regard to children, first, because most of the questions in the WG-SS are not suitable for children under the age of 5 years, and second, because the WG-SS leaves out important functional domains for children aged 5–17 years, namely those related to developmental disabilities and behavioural issues.

Additionally, feedback on the administration was found to be very positive, and interviewers reported no challenges with its implementation (see Annex B). In Serbia, for example, caregivers of children with functional difficulties reported that they preferred the CFM to the questionnaire that included the TQSI and WG-SS as it provided a deeper and more sensitive evaluation of their children’s difficulties.