Potential Selective Responding in a Parent Questionnaire Study of Post-Institutionalized Children

Abstract

Selective responding bias, though under-researched, is of particular concern in the study of post-institutionalized children because many studies rely on mailed questionnaires and response rates are often low. The current study addresses the impact of selective responding in a single wave of data collection and in a multi-wave study. Participants were 121 parents from a larger four-wave study of post-institutionalized children, identified as Never Responders, Previous Responders (but not to the current wave), or Wave 4 Responders. Parents were telephoned and asked about their adopted child's family, school, peer, and behavioral adjustment. The children (47% male) ranged in age from 2 to 20 years (M = 10.79, SD = 4.59) and had been adopted between 5 and 54 months of age (M = 15.49, SD = 9.94). There were no differences in parent ratings of adjustment for a single wave of data collection; however, participants who never responded reported poorer family and peer adjustment than those who had responded to at least one wave of data collection. Within a single wave of data collection, there was no evidence that selective responding contributes much bias. Over a multi-wave study, however, results may under-represent adjustment difficulties, especially with family and friends.

One of the most commonly used methods of data collection in research involving post-institutionalized (PI) children is a questionnaire that relies on informant reports of child behavior and/or abilities. Although there are some comprehensive longitudinal studies employing direct observational assessments of PI children (e.g., Ames, Chisholm, Fisher, Morison, Thompson, & Mainemer, 1997; Rutter et al., 2010), a substantial segment of this literature is based on parent-reported assessments of PI children's behaviors (e.g., Fisher, Ames, Chisholm & Savoie, 1997; Groza, 1999; Gunnar, van Dulmen, & the International Adoption Project Team, 2007; Hawk & McCall, 2011; Merz & McCall, 2011). Furthermore, most of these studies rely on mailed questionnaires, which parental respondents must return. This situation is due mainly to the limited availability of adoptive families within a specific location and the prohibitive cost, both monetary and time, of conducting direct observations on hundreds of participants. Such questionnaire studies have found more behavior problems including inattention (Fisher et al., 1997; Groza, 1999; Gunnar et al., 2007; Hawk & McCall, 2011; Hoksbergen et al., 2004), executive functioning deficits (Merz & McCall, 2011), and social difficulties (Le Mare, Warford, & Fernyhough, 2001) among PI children relative to non-adopted, parent-reared children.

These parent-reported questionnaires provide a number of advantages over direct observations. First, scores on questionnaires potentially represent longer and more diverse observation periods because parents experience a target child over a long period of time and often in different situations, some of which are not accessible to observers. Second, they represent a type of “social ecological validity” because they reflect behaviors in the target child that are noticeable to others and may be of social and personal significance. Third, people tend to act on the basis of their perceptions. Thus, if a parent perceives a child as being “aggressive,” that perception – perhaps more than the child's actual behaviors – is likely to influence the informant's attitude and behavior toward the target child, potentially influencing the child's behavior in that setting.

However, a main concern of mailed questionnaire research is that certain types of informants (e.g., higher socio-economic status, those with problematic children) may be more likely to respond than others. This selective responding bias could have a substantial influence on results in cases in which only a small percentage of contacted individuals actually return their questionnaires. Although various attempts have been made to encourage as large a percentage of parents to respond as possible, response rates in the PI literature have varied substantially, from at least as high as 65.6% (Gunnar et al., 2007) to at least as low as 24% (Groza, 1999). Self-selected sample bias, which influences which families do and do not return their questionnaire, could have a relatively large effect on results, especially when response rates are low.

For example, it is reasonable to speculate that parents who have PI children who are adjusting well over the years in their adoptive families tend to take the time to fill out a questionnaire and to return it than are parents of PI children who are experiencing difficulties. Parents enjoy talking about their children, and parents of PI children may be especially disposed to doing so because they want to counter the image produced by a few celebrated cases of problem children and unsuccessful adoptions (e.g., Friess, 2005). Further, parents of children who are having adjustment problems (e.g., behavioral issues, cognitive deficits, social problems) may be inhibited from responding out of concern that they might not be granted another child if the adoption agency or others responsible for arranging adoptions would come to know of their particular issues, despite assurances of confidentiality by researchers. Also, some may feel personally responsible for their child's issues and not want to review them on a questionnaire. Such responding bias would result in under-reported problems and, possibly, false-negative findings.

Conversely, it is also reasonable to hypothesize that adoptive parents of PI children who are having adjustment problems might be more likely to respond to questionnaires than those whose children are adjusting very well. Most such questionnaires do focus on problem behaviors, and parents whose children are adjusting well occasionally report that they do not fill out the questionnaire because their child does not have these problems. On the other hand, parents of children who are having somewhat severe behavioral issues may be quite eager to tell someone they perceive as an authority (i.e., the researcher who is likely to be a University-based psychologist or other professional) of the difficulties they and their child are having in hopes that something can be done in the future for others if not for them. This bias would result in an inflated estimate of PI difficulties and the potential for false-positive findings.

Finally, longitudinal designs are frequently used to assess the development of PI children in the adoptive home. In these studies, which mail questionnaires in waves to assess children periodically, parents may respond to some waves but not others. Thus, if the researchers use only data from a single wave, they must understand the differences between responders and nonresponders for that wave to assess selective responding. However, if they collapse data across waves to increase power by increasing the number of participants, they must assess selective responding by comparing those who responded to at least one wave (forming their sample) and those who never responded to the questionnaire.

To the best of the authors’ knowledge, no studies of such selective sampling bias have been conducted with PI children. Of course, conducting an evaluation of sampling bias is made difficult by the fact that parents who choose not to return the questionnaire do not provide information about the adjustment of their children to be compared with the adjustment of children whose parents actually do return the questionnaire. A potential solution to this methodological conundrum is provided by Goodman (1999), who observed that responses to a single question, “Is there a problem?” were almost as good at discriminating between clinic and community control samples as were complete impact scores from the Strengths and Difficulties Questionnaire. This result provides a rationale for simply calling parents who did and did not return questionnaires and asking a few simple questions regarding the general adjustment of the child.

The current study addresses the issue of selective responding to mailed questionnaires by assessing the adjustment of PI children whose parents never responded to up to four waves of data collection, previously responded but not to the most recent wave, and responded to the most recent wave. Parents were contacted by telephone and asked simple questions about their child's general adjustment in regard to family, school, peers, and behaviors. This design allowed the analysis of two distinct but important questions: 1) In a given wave of data collection, are nonresponders different from responders, and 2) in a multi-wave study, are parents who never responded different from those who responded at least once?

Method

Participants

Participants were part of a larger pool of subjects in a four-wave study of post-adoption outcomes of Russian orphanage children adopted through an agency specializing in adoptions from the Russian Federation. This parent study has sent four waves of questionnaires to parents over a period of 10 years (2001, 2003, 2008, 2010) and contains at least one returned questionnaire from 820 parents. At each wave, questionnaires were sent to all families that had a child adopted from Russia (ages 2-20 years) and had adopted through a single adoption agency. Respondents to the current study were 121 parents of children adopted from Russian orphanages. The children (47% male) ranged in age at the time of assessment from 2 to 20 years (M = 10.79, SD = 4.59), and they had been adopted between 5 and 54 months of age (M = 15.49, SD = 9.94). Although questions were not asked about ethnicity or socio-economic status (SES), the large majority of respondents to the parent study are Caucasian and of high SES.

The questionnaires in the parent study consist of family and child demographic information, questions about the pre-adoption environment, and four to six surveys about the child's behavior. They take approximately an hour to complete, and respondents were offered $30 at each wave. From the list of families who were mailed questionnaires in the most recent wave of data collection (wave 4), subjects were identified as Non-Responders (i.e., those who had never responded to any wave of data collection), Previous Responders (i.e., those who had responded to one to three previous waves but not wave 4), and Wave 4 Responders (i.e., those who had responded to wave 4, including some who had not previously responded). Each group was also stratified for child age (0-10 years and 11-20 years). The fourth wave was chosen because it was the most recent and, thus, most appropriate for a follow-up survey.

A stratified sampling strategy was used in which, using a random number generator, subjects were randomly selected from the larger pool of possible subjects within each responder-by-age group (i.e., 0-10-year-old Never Responders, 11-20-year-old Never Responders, etc.). Because some phone numbers were disconnected or changed, phone calls were made to the selected individuals until approximately 20 subjects within each group were reached (i.e., 40 participants per responder group). Of those contacted, six refused to participate (all nonresponders), two were outside of the age at assessment and age at adoption limits (1 previous responder; 1 wave 4 responder), and five failed to answer every question (2 non-responders; 1 previous responder; 2 wave 4 responders). Thus, the final sample size was 108.

Procedure

Contacted parents were asked to answer a few questions about their oldest child adopted from the Russian Federation. Participants were asked for their child's current age, age at adoption, and gender. They were also asked four simple questions: how has your child adjusted to your family life? (2 = well, 1 = average, 0 = poorly), how has your child adjusted to school and schoolwork? (2= well, 1 = average, 0 = poorly), how is your child's relationship with peers? (2 = above average, 1 = average, 0 = below average), and would you say that your child has fewer, about the same, or more behavioral problems than most children his/her age? (2 = fewer, 1 = about the same, 0 = more). Thus, higher scores represented better adjustment.

Results

Preliminary Tests of Covariates

All analyses were designed to examine the two main questions: 1) Wave 4 Responders versus Wave 4 Non-Responders (composed of Non-Responders + Previous Responders) represented a test of responder vs. non-responder on a single wave of data collection. 2) Never Responders versus Ever Responders (composed of Wave 4 Responders + Previous Responders) represented a test of responders vs. non-responders across all four waves of data collection (e.g., for a possible repeated measures longitudinal study). Past research has suggested that age at assessment, age at adoption, and gender may be related to specific adjustment problems in PI children (e.g., Groza, 1999; Hawk & McCall, 2010; Rutter et al., 2010); therefore, these potential parameters were investigated as possible covariates for the main analyses. To determine whether this demographic information should be included as covariates, two sets of analyses were performed. The first analysis determined whether the responder groups differed on demographics. For the Wave 4 Responders vs. Wave 4 Non-Responders, neither age at assessment, F(1, 111) = 1.54, p = .22, partial η² = .01, age at adoption, F(1, 111) = .01, p = .94, partial η² < .001, nor gender, χ²(1) = .17, p = .68, were associated with responder type. Similarly, neither age at assessment, F(1, 111) = .56, p = .46, partial η² = .01, age at adoption, F(1, 111) = 3.47, p = .07, partial η² = .03, nor gender, χ²(1) = .20, p = .65, were associated with the Ever vs. Never Responder comparison (see Table 1).

Table 1.

Mean (Standard Deviation) Demographics and Adjustment Ratings for Each Responder Type

	Single Wave		Multi-Wave
	Wave 4 Non-Responders (N =71)	Wave 4 Responders (N = 37)	Never Responders (N = 32)	Ever Responders (N = 76)
Age at Assessment (Years)	11.18 (4.57)	10.05 (4.60)	11.28 (5.23)	10.58 (4.31)
Age at Adoption (Months)	15.55 (9.83)	15.38 (10.29)	18.12 (11.08)	14.36 (9.26)
Gender (% Male)	46%	50%	44%	49%
Family Adjustment	1.77 (.57)	1.89 (.39)	1.59 (.71)	1.91 (.37)a
School Adjustment	1.45 (.73)	1.46 (.73)	1.25 (.76)	1.54 (.70)
Peer Adjustment	1.18 (.72)	1.35 (.63)	0.94 (.76)	1.37 (.63)a
Behavior Problems	0.96 (.64)	1.00 (.67)	0.87 (.75)	1.01 (.60)

^ap < .05

The second set of analyses investigated whether the demographic information was related to the outcome variables. Regression analyses with age at assessment, age at adoption, and gender as the independent variables were run predicting the four outcome variables. The model predicting family adjustment was significant, F(3, 107) = 4.57, p = .005, R² = .11. Parents of children older at assessment reported poorer family adjustment (B = -0.03, SE = 0.01), t(107) = 2.46, p = .02. Somewhat similarly, the model predicting school adjustment was marginally significant, F(3, 105) = 2.68, p = .05, R² = .07, with parents of older children reporting poorer school adjustment (B = -0.03, SE = 0.02), t(105) = 1.98, p = .05. However, neither the model predicting peer relationships, F(3, 105) = 0.29, p = .83, R² = .01, nor behavior problems, F(3, 105) = 0.79, p = .50, R² = .02, was significant. Based on these findings, only age at assessment was included as a covariate in the main analyses.

Selective Responding

Two multivariate analyses of variance (MANOVAs) were computed on family adjustment, school adjustment, peer relationships, and behavior problems to examine whether selective responding in a single wave (question 1) or in any wave of a multi-wave study (question 2) were related to these outcomes.

The first analysis, Wave 4 Responders versus Wave 4 Non-Responders, did not have a significant overall MANOVA, F(4, 102) = 102.00, p = .66, partial η² = .02. Neither family adjustment, F(1, 105) = .75, p = .39, partial η² = .01, school adjustment, F(1, 105) = .05, p = .82, partial η² = < .001, peer relationships, F(1, 105) = 1.24, p = .27, partial η² = .01, nor behavior problems, F(1, 105) = .11, p = .74, partial η² = .001, were related to wave 4 participation (see Table 1).

The second analysis, Never Responders versus Ever Responders, however, had a significant overall MANOVA, F(4, 102) = 3.61, p = .009, partial η² = .12. Responders reported better family adjustment, F(1, 105) = 8.31, p = .005, partial η² = .07, and peer relationships, F(1, 105) = 8.94, p = .003, partial η² = .08, than Never Responders (see Table 1, Figure 1). They did not differ on school adjustment, F(1, 105) = 3.11, p = .08, partial η² = .03, or behavior problems, F(1, 105) = 1.04, p = .31, partial η² = .01 (see Table 1, Figure 1).

Figure 1.

Mean adjustment scores for the a priori contrast of Never Responders vs. Ever Responders. The contrast was significant for family and peer adjustment (p < .05) but not school or behavioral adjustment.

Discussion

Summary

In this random sample of parents who did and did not respond to a mailed survey about their PI adopted child, two main outcomes were observed. First, within a single wave of data collection, those who chose to respond did not differ from those who did not respond. Second, over the course of a multi-wave study (up to four waves), those who chose never to respond reported poorer family adjustment and peer relationships than those who had responded at least once.

Single Wave of Data Collection

The first finding fails to provide significant evidence to indicate that selective responding bias with respect to children's adjustment occurred for a single wave of data collection. This non-significant result plus the low percent variance associated with respondent group suggests that respondent selectivity is not a large concern in a single wave of data collection.

At a given point in time, many factors may contribute to a parent's choice to respond or not. The timing of the questionnaire may be particularly problematic for the parent, resulting in either the parent's forgetting or choosing not to complete it. The questionnaire was somewhat long and demanding, and parents may simply have decided to complete it later, with later never arriving. This type of problem would presumably affect parents with problematic and unproblematic children equally. Conversely, the child's adjustment may also have influenced responding, with parents of particularly problematic children not wanting to divulge this information and with parents of particularly well-adjusted children not feeling the need to respond. On the other hand, the reverse may also be the case. Some parents of problematic children are eager to return questionnaires in the hope of receiving help or making help more feasible in the future; whereas parents of non-problematic children want to tell others about how well their children have adjusted. For a single wave of data collection, these dispositions, to the extent that they occur, seem to cancel each other out. Thus, we have no evidence that selective responding as a function of child adjustment problems influences to a noticeable extent which parents respond to a single wave of data collection.

Multiple Waves of Data Collection

According to the second finding, however, selective responding may become an issue over the course of a multi-wave study. Although response to any single wave seems to be within the realm of random, parents who continually decide not to respond may be more likely to have children with family and peer adjustment problems than those who respond to at least one wave of data collection. Even then, the percent variance associated with responder group was only 12%.

This phenomenon seems to support the hypothesis that at least some parents of children with problems, especially regarding family and peer adjustment, selectively choose not to participate in studies assessing their children's adjustment, despite assurances of confidentiality. This may be for a number of reasons, such as fear of not being granted another child or feelings of guilt or embarrassment. Children with adjustment problems may also cause their families to be more involved in other support activities (e.g., counseling, tutoring), which may result in less time to devote to a questionnaire. Some of these children have severe behavioral issues and are overwhelming to their parents, who may not want to revisit the details in the course of filling out a questionnaire or who are too occupied with their children to complete the questionnaire. Regardless of the reason, studies that rely on repeated assessments of the same families may underestimate the number or severity of adjustment problems because of a reticence of parents with children with problems to respond to any of several waves of data collection.

Conclusion

These findings suggest that when studies of PI children present findings stemming from a single wave of data collection, the results can be interpreted with limited concern for selective responding bias. However, when these studies use data from multiple waves of data collection, the findings may under-represent problems in the total population, especially with respect to family and peer adjustment. Because these studies often report elevated levels of behavior problems among PI children, especially those who are adopted later in infancy (Fisher et al., 1997; Groza, 1999; Gunnar et al., 2007; Hawk & McCall, 2010, 2011), the fear is not that they are over-representing the difficulties of this population but that they may not be assessing the full extent of behavior problems. Further, the outcome variable made a difference in the current findings. Ratings of school adjustment and behavior problems were not significantly different overall or in either a priori comparison. Thus, the potential for selective responding is apparently not uniform across adjustment problems in different realms. Rather, it seems to be a concern specifically for parental perceptions of family and peer adjustment.

Limitations

The current study may be among the first to assess the possibility of selective responding with respect to children's adjustment problems in a large, multi-wave assessment of PI children. However, some limitations exist. First, most parents, including those who had not returned the questionnaires, did respond to the telephone interview, but a few did not. Only six of 121 refused to participate fully, but all were in the Non-Responders group.

Second, adjustment measures consisted of only one question, answered in a Likert format with three possibilities. This type of assessment does not give the same degree of information as a longer assessment with a more nuanced scale. Goodman (1999), however, has found that a single question, such as those in the current study, was nearly as effective as a more comprehensive assessment at distinguishing between a clinic and community sample. Further, many of the commonly used questionnaires have a three-answer format for individual items, consistent with the current use of only three options.

This study is also limited by the fact that one of the main conclusions, no selective responding for a single wave of data collection, is based on accepting the null hypothesis. This technically does not disprove that hypothesis but fails to provide evidence sufficient to reject it. Of course, if the sample size were progressively increased, eventually the hypothesis would be rejected. However, only 2% of the overall outcome variance was associated with responder vs. non-responder in the single wave test. This finding suggests that the size of the effect is relatively small regardless of sample size. Also, the means in Table 1 varied between .87 and 1.91 for a 0-2 scale with standard deviations of approximately .70 (95% confidence interval of +1.40), indicating that restricted ranges of responding were not a constraining factor. Although these limitations are important considerations, the current study offers a first look at this important question and suggests that selective responding may not be as large a problem as some have supposed, especially for a single wave of data collection.