Maintaining a Social-Emotional Intervention and its Benefits for Institutionalized Children

Abstract

This paper reports the maintenance of one of the largest interventions conducted in St. Petersburg (Russian Federation) orphanages for children birth to 4 years using regular caregiving staff. One orphanage received training plus structural changes, another training only, and a third business as usual. The intervention produced substantial differences between these institutions on the HOME Inventory and on the Battelle Developmental Inventory scores for children. These institutional differences in HOME scores (N=298) and Battelle scores for children (N=357) departing the institutions for families in St. Petersburg and the USA were maintained for at least six years after the intervention project, result may be associated with to certain features of the intervention and activities during the follow-up interval.

Sustainability refers to maintaining the effectiveness of interventions and transferring them to community agencies to be operated after the demonstration funding and researchers are no longer involved. The inability of researchers and practitioners to sustain successful interventions in the community has been lamented for decades. For example, Sarason (1967) noted that “psychologists are as good as anybody else in initiating change and as bad as everybody else in sustaining it” (p. 232). This frequent lack of success is a long-standing issue in many domains, including public health (e.g. Altman, 1995; Goodman & Steckler, 1987, 1989a, 1989b), the prevention of adolescent problem behavior (e.g., Bumbarger & Perkins, 2008; Gomez, Greenberg, Feinberg, 2005; Tibbits, Bumbarger, Kyler, & Perkins, 2010), and social-emotional and early care and education interventions for young children (e.g., Domitrovich, Moore, & Greenberg, 2011). Of course, some interventions have been sustained (e.g., Eckenroade et al., 2010), but it is still often the case that intervention effects commonly fade after initial funding terminates, the intervention is transferred from researchers to practitioners, and attempts to implement the original program in new sites and new communities are made (e.g., Gomez et al., 2005; Hallfors, Cho, Livert, & Kadushin, 2002; Hallfors & Goddette, 2002; Mancini & Marek, 2004).

Across various disciplines, “sustainability” refers to a set of functions, typically conducted in sequence, including 1) demonstrating that the original intervention was effective, 2) maintaining the intervention effects in the original setting after initial research funding ends, 3) transferring responsibility for the intervention to practitioners and community agencies, 4) getting the same or similar evidenced-based interventions used in new communities by non-researchers, and 5) demonstrating over long periods of time the effectiveness of such evidence-based interventions in communities. Generally, these literatures converge on the proposition that better long-term outcomes are obtained if planning for sustainability occurs when the original intervention is created, and if steps are taken along the way that are deliberately aimed at facilitating the maintenance of the intervention and its transference to new contexts and practitioners (e.g. Altman, 1995; Berry, Senter, Cheadle, Greewald, & Peason, 2005, Mancini & Marek, 2004; Mittelmark, Hunt, Heath, & Schmid, 1993; Scheirer, 2005).

This paper focuses on the second of the above sustainability components, which is called “maintenance” of effectiveness. Once funding for the initial demonstration has ended and the intervention becomes a routine part of an institution’s programming, the fidelity with which the intervention is implemented often deteriorates, and the benefits once demonstrated diminish with time, sometimes completely. Typically this occurs when no plans or activities were made for intervention maintenance.

This has been the case for interventions designed to improve the development of institutionalized children. Most of the early interventions were conducted primarily as basic research projects designed to show increased stimulation of different kinds would improve children’s development. These studies provided infants and young children with specific extra stimulation implemented by the experimenter or research assistants for a short period of time each day over several weeks. Generally, such interventions produced developmental improvements or they prevented the developmental decline that was observed in untreated infants. These were “demonstrations” that benefits could be produced, there were no attempts to continue the intervention, and thus the developmental benefits diminished or disappeared (Bakermans-Kranenburg, van IJzendoorn, & Juffer, 2008; Rosas & McCall, in press). Even more comprehensive and longer-term demonstration interventions may produce fading benefits if nothing is done to deliberately maintain them. For example, Sparling, Dragomir, Ramey, and Florescu (2005) provided young Romanian institutionalized children with newly hired and trained caregivers, one for each of four children, who conducted a variety of educational activities over a 12-month intervention period. The children improved on the Denver Developmental Screening Test, but Carlson and Earls (1997) reported that the benefits to these children and to the institution faded in the years following the end of the intervention.

More recently, some interventions have been aimed at transforming an entire institution using regular staff with a long-term goal of producing permanent improvement in its operation and children’s development. An estimated 2 (USAID, 2009) to 8 million (Human Rights Watch, 1999) children reside in institutions worldwide, and the published literature indicates that conditions in most institutions are not supportive of children’s development (e.g., McCall, Van IJzendoorn, Juffer, Groark, & Groza, 2011; Rosas & McCall, in press). Although most international child welfare professionals advocate for family alternatives to institutions, it is likely to take many years, if not decades, for most low-resource countries to place all children without permanent parents in families. Consequently, institutions potentially could be improved for the many children who are likely to reside there while alternatives are being developed (Groark & McCall, 2011; McCall, 2012).

The current study is a follow-up of the most comprehensive intervention reported in an institution for infants and young children using regular staff. It was conducted in 2000–2005, with follow-up assessments extending six years after the end of the original intervention project. Three institutions (called Baby Homes, BHs) in St. Petersburg (Russian Federation) were involved. In one BH, caregivers received training designed to promote more warm, sensitive, contingently-responsive, and child-directed caregiver-child interactions; and a variety of structural changes were implemented to support the training and promote social-emotional relationships between caregivers and children in a more family-like context. A second BH received training but no structural changes, and a third continued with business as usual. The intervention produced the intended changes in caregiver behavior and the BH environment as measured by the HOME Inventory; and children’s physical, cognitive, and social-emotional development improved substantially in the double intervention BH (The St. Petersburg-USA Orphanage Research Team, 2008).

In addition, the intervention was designed to be consistent with the sustainability literature (reviewed above) and its specific suggestions for maintaining interventions (Groark & McCall, 2008; The St. Petersburg-USA Orphanage Research Team, 2008). For example, the intervention was designed to be maintained after the demonstration phase on the regular government-supplied budget of the BH, and additional steps were taken after the end of the intervention project to maintain its effectiveness (see below).

It was hypothesized that these maintenance activities would preserve the intervention in two ways, first by maintaining the improved caregiver behavior and BH environment as reflected on the HOME Inventory, and second, as a result, maintaining the benefits to children’s behavioral development as measured by the Battelle Developmental Inventory.

A follow-up project was funded approximately two years after the intervention project had terminated. The follow-up focused on a subset of children from the three intervention BHs, including children adopted to the United States through the International Assistance Group, a Pittsburgh-based adoption agency specializing in the placement of Russian children in USA families, and children placed in St. Petersburg adoptive, foster, and birth parent or kinship families. Children departing the three BHs for these family destinations were given some of the same measurements as during the intervention. This study uses this reduced set of measurements on both the intervention and follow-up samples of children transitioning to the above destinations.

To the best of the authors’ knowledge, this is the only long-term follow-up of a comprehensive intervention using regular institution staff. Follow-up assessments of children randomly assigned either to remain in the institution or go to a specially-designed foster care program have been reported (e.g., Fox, Almos, Degnan, Nelson, & Zeanah, 2011; Johnson et al., 2010). Children who remained in their randomly assigned conditions through 8 years of age have continued to demonstrate physical and cognitive benefits of foster care. Presumably, the quality of foster care remained high following placement, so the “treatment” condition persisted, but limited assessments of its continuing quality have been published. Some children were transferred to government organized foster care, which tended to be somewhat less effective (Fox et al., 2011).

Method

Participants

Baby Homes (BH)

This study was conducted in three Baby Homes (BH) caring for children birth to 4 years of age in St. Petersburg (Russian Federation) which had been involved in a major intervention project from 2000–2005. Details of these institutions, the caregivers and children involved, the interventions that were implemented, and the results of those interventions can be found in The St. Petersburg-USA Orphanage Research Team (2005, 2008).

Briefly, caregivers in one BH received training in infant-toddler development and in warm, sensitive, contingently-responsive, child-directed caregiver-child interactions. In addition, numerous structural and employment changes were made in this institution that promoted better caregiver-child interactions from fewer caregivers. This condition was called Training + Structural Changes (T+SC). The structural changes consisted of 1) smaller group sizes, 2) two primary and four secondary caregivers assigned to each subgroup in which one of the primary caregivers was present for most of the children’s waking hours seven days a week, 3) a reduction from approximately nine to six caregivers assigned to a ward but minimum change in total caregiver hours, 4) heterogeneous grouping of children by age and disability status within a group, 5) termination of periodic graduations of children to new wards when they reached a certain age or developmental milestone, 6) the regular assignment of specific caregivers to be substitutes for a particular ward when needed, 7) the creation of an in-house monitoring and supervisory system, and 8) family hour in both the morning and afternoon in which visitors to the wards were not allowed and children were to be with their caregivers. A second BH received Training Only (TO), and a third conducted business as usual and had No Intervention (NoI).

T+SC was implemented first, and it took approximately one year to complete the training and structural changes. Then training started in TO, and assessments were begun in NoI shortly thereafter. The T+SC intervention was completely implemented by 3 September 2000, the TO intervention was completed on 13 January 2002, and assessments were begun in NoI on 18 October 2002.

Maintenance activities

Attempts to maintain the intervention were of two types–creating an intervention that had features that would promote maintenance after the original demonstration project ended and procedures implemented during the intervention and the time following the intervention that might contribute to program maintenance.

The original intervention involved several components thought to promote maintenance. First, the double intervention was implemented in a BH in which the director was thoroughly committed to implementing and maintaining the intervention and insisted that her staff do so. This director remained in her position throughout the intervention and follow-up periods.

Second, although the intervention project provided substantial resources to this BH including some funds to hire additional caregivers, the intervention was designed to reduce the number of different caregivers in children’s lives but not to increase the number of caregiver hours available to children. Thus the total number of caregiver hours available in the BH did not change much with the intervention and could be sustained financially in large part after intervention funds terminated.

Third, a train-the-trainer strategy was adopted so that training personnel were continually available to prepare new caregivers who replaced those who left during and after the intervention.

Fourth, an in-house monitoring and supervision system was established in which specialists (e.g., staff professionals in children with disabilities, early education) were responsible for monitoring the caregivers and encouraging them to implement the training on the wards on a continuing basis.

Once the intervention project ended, all financial support stopped, and the interventions had to be maintained on the BHs’ government-provided budget. In the Training Only institution, separate funds were obtained after the intervention period to assist this BH to implement structural changes, especially reduced group size. This was only partly successful, because not all groups were reduced in size, integration by age and disability status was only partly accomplished, and family hour was not implemented.

Further, following the end of the intervention project, additional coaching and technical assistance were provided by author Palmov and two other professionals to both T+SC and TO, which consisted of weekly or bimonthly visitations, supervision, and periodic refresher training. Also, author Groark visited caregivers in T+SC and TO two or three times per year to encourage appropriate caregiver-child interactions and troubleshoot. These supports were given during and between both projects. When several new caregivers were hired in T+SC and TO, a brief training course consisting of 25 hours over 4–5 weeks was given during and after the intervention period by those specialists originally trained as “trainers.” However, the in-house monitoring and supervisory system encouraging appropriate caregiver-child interactions that had been implemented in the T+SC was not created in TO.

Caregivers

The primary and secondary caregivers in T+SC and the medical nurses and assistant teachers (i.e., main regular caregivers) up to approximately 50 in each BH were individually assessed periodically with the HOME Inventory (24-month group version; Bradley & Caldwell, 1995; Caldwell & Bradley, 1984).

The number of caregivers holding these positions varied from one assessment occasion to the next and between BHs. At each time point, 46–59 caregivers in T+SC, 45–53 caregivers in TO, and 30–52 caregivers in NoI were assessed. It should be noted that NoI was reduced substantially in size by the local government by the time the follow-up project assessments were conducted, which explains its lower number of caregivers during the two follow-up assessments.

Departing children

The sample for the current analyses was dictated by the sampling criteria for the follow-up project which focused on assessing the development of children who transitioned from these three BHs to families, either in the St. Petersburg region or to the United States. This sample (see Table 4 for Ns) consisted of all children who were being adopted into the United States (USA adoption) under the auspices of the International Assistance Group, a Pittsburgh adoption agency specializing in placing children from St. Petersburg and other cities in the Russian Federation. In addition, all children who were being returned to birth parents or taken by relatives of the birth parent (i.e., birth family), entering foster care (i.e., foster care; foster parents are paid monthly by the government), and adopted or being taken by “non-relative kin” (a classification of parents who receive some financial assistance and may or may not adopt the child if and when parental rights are legally terminated) in the St. Petersburg area (collectively St. Petersburg adoption). Only adopted children who spent at least 3 mos. in residence in one of the three BHs described above after the interventions had been completely implemented were included, a residency requirement deemed the minimum time necessary for the BH interventions to have an effect on children. Their average length of residency was 12.8 mos. (SD=8.9) Therefore, this sample does not include children who “passed through” a BH in less than 3 mos., did not transition to families, were adopted to Russian families living outside St. Petersburg, adopted to the USA through other agencies, adopted to other countries, or aged out of the institution. Consequently this sample included fewer children with disabilities or other developmental problems than the total BH population. No children were sampled between the intervention and follow-up project periods because of lack of funding.

Table 4.

Departure Battelle Subscale DQ Means (SDs) as a Function of Intervention Conditions and Time

Intervention Condition BDI Subscale	Intervention	Follow-Up Time 1	Follow-Up Time 2	Total
T+SC(N=46,59,42;147)
Personal-Social	77.8(19.7)	75.9(25.9)	79.7(20.1)	77.6(22.4)
Communication	81.4(24.0)	75.2(23.2)	83.9(20.2)	79.6(22.8)
TO(N=61,31,19;111)
Personal-Social	65.9(17.5)	81.3(23.9)	79.8(24.9)	72.6(21.9)
Communication	67.0(15.1)	80.8(22.1)	80.3(23.1)	73.1(19.8)
NoI(N= 43,43,13; 99)
Personal-Social	61.4(17.2)	71.5(21.5)	70.4(21.4)	67.0(20.1)
Communication	58.9(17.0)	65.3(18.9)	67.4(24.1)	62.8(19.0)
Total (N= 147,111,99;357)
Personal-Social	68.3(19.2)	75.7(24.2)	78.1(21.7)	73.1(22.0)
Communication	69.1(20.6)	73.3(22.3)	80.1(22.2)	73.0(21.9)

Departing children fell into three time periods: Those who departed 1) during the intervention project, 2) during the first two years of the follow-up project (Follow-up Time 1) and 3) during the third and fourth years of the follow-up project (Follow-up Time 2). Ns varied with time period, BH, and assessment and are reported with the results below.

Assessments

HOME Inventory

The 24-month institutional version of the HOME Inventory (Caldwell & Bradley, 1984; Bradley & Caldwell, 1995) was selected because this age was approximately in the middle of the age range of children residing in these BHs. It consists of 5–11 items in each of six subscales (responsivity, acceptance, organization, learning materials, involvement, and variety) that are scored “yes” or “no.” Subscales and total scores represent the number of items scored “yes.” Notice that a caregiver needs to display a given item only once to at least one child to be given a “yes” for that item. Thus, the HOME reflects the presence of certain characteristics but generally not their prevalence within a ward. In addition, a special sociability subscale was created by the authors consisting of the number of yes’s to all items on the HOME that pertained to the social behavior of caregivers and children. Specifically, the sociability scale included 21 items (e.g., caregiver continuously vocalizes to children at least twice, caregiver responds verbally to children’s vocalizations or verbalizations, caregiver converses freely and easily, caregiver invests maturing toys with value via personal attention, caregiver structures children’s play periods, etc.; see The St. Petersburg-USA Orphanage Research Team, 2008).

The HOME Inventory was conducted focusing on an individual caregiver attending to 5–14 children, typically with other caregivers present. An assessment consisted of 60 min. of observation, including at least 45 min. in which the children were not asleep and not being fed, changed, or bathed (i.e., “free time”), plus 10–15 min. in which they were engaged in feeding, changing, or bathing.

The HOME is one of the most widely used assessments of the environment and caregiver behavior in home and non-residential group settings, and the total score and subscale scores have been found to correlate with motor, social, and mental competence in young children (Bradley, Corwyn, Burchinal, McAdoo, & Coll, 2001), to predict mental performance in adolescents (Pettit, Bates, & Dodge, 1997), and to be sensitive to preventive early interventions (Bakermans-Kranenburg, van IJzendoorn, & Bradley, 2005).

Battelle Developmental Inventory

The Battelle Developmental Inventory (LINC Associates, 1988) was used to assess children’s general behavioral development. It was selected because the items were more relevant (“authentic”) to the BH context and it was better suited to children with mild disabilities than many other tests of general behavioral development. It is appropriate for children birth to 95 mos., and provides a total score plus subscales for adaptive behavior (i.e. self-maintenance skills), gross motor, fine motor, communication, cognition, and personal-social development (see details in The St. Petersburg-USA Orphanage Research Team, 2008). However, while the full scale was administered during the intervention, only the communication and personal-social subscales were given at departure during the follow-up project and reported here, because the intervention produced the maximum differences between BHs on these two subscales. Scores are reported as Developmental Quotients (DQs), which consisted of mental age divided by chronological age.

Procedure and Reliability

HOME

The HOME was administered in an entire BH before any training or structural changes were begun (i.e., pre-intervention), approximately a year after the interventions were completely implemented in a BH, and every year thereafter until the end of the intervention project (2005). Approximately two years later, the follow-up project was initiated, and the HOME was administered again in approximately 2008 and 2011.

Only the pre-intervention HOME assessment, the last two assessments (i.e., Intervention Time 1 and Time 2) during the intervention period which occurred in approximately 2003 or 2004 and 2005, and the two follow-up assessments Follow-up Time 3 and Follow-up Time 4 (in 2008 and 2011) are reported here. The actual time between assessments varied somewhat between BHs, so the length of time between assessments is approximate. Also, HOME assessments given to replacement caregivers between these all-BH assessments during the intervention project are not utilized in this report, which means that some data presented here are not identical to similar data presented in The St. Petersburg-USA Orphanage Research Team (2008).

A team of four assessors was trained by two experts in the HOME. After practicing, formal reliability was determined before the intervention project and approximately two years later. Correlations between assessors were .90+ for the six subscales and .98 for the total score. Agreement was slightly lower two years later in part because of lower variability in the reliability sample (for details, see The St. Petersburg-USA Orphanage Research Team, 2008). The same team of assessors (minus one who left after the intervention project) conducted all of the HOME assessments across the more than 10 years of this project.

Battelle Developmental Inventory

Battelle’s were administered within the BH approximately one month before children’s departure (M= 4.6 weeks, SD=7.1).

During the intervention project, children were comprehensively assessed periodically, but only data from the Battelle personal-social and communication subscales from their departure assessment are reported here (i.e., Intervention). During the follow-up project, departure scores obtained during the first two (i.e., Follow-up Time 1) and next two years (i.e., Follow-up Time 2) are reported here.

Four Battelle assessors were trained in the administration of the Battelle, and two years into the intervention project two additional assessors were trained. Formal reliabilities were obtained for all assessors. For the total score, 87% of the pairs of scores during formal reliability were within two points, suggesting that unreliability was less than 2% of the average subscale score and approximately 1% of the average total score (The St. Petersburg-USA Orphanage Research Team, 2008).

Results

Estimates of Staff Turnover

The HOME data provided the opportunity to estimate staff turnover. Turnover of staff represents a potential impediment to maintaining the intervention, it contributes to inconsistencies of caregivers in children’s lives and thus could influence children’s developmental scores, and it could be a confounding factor if the three BHs were markedly and persistently different in turnover rates.

Table 1 presents estimates of staff turnover in three time periods covering eight years in the three BHs in this study. The first time period is approximately 2003–2005, which represents the last 18 months of the intervention project. The second time period is approximately 2005–2008 representing the three years between the final HOME assessment of the intervention project and the first assessment of the follow-up project. The third time period is approximately 2008–2011 representing the three years between the two assessments conducted during the follow-up project.

Table 1.

Two Estimates of Staff Turnover Over Eight Years in Three Baby Homes

BH	2003–2005 = 18 mos.		2005–2008 = 3 yrs.		2008–2011 = 3 yrs.		Wt.’d Ave.
	T1–T2a	T2b	T2–T3a	T3b	T3–T4a	T4b	Annual Ratec
T + SC	10/46	10/56	29/56	29/50	17/50	17/49
	22%	18%	52%	58%	34%	35%
Annuald	11%	9%	17%	19%	11%	12%	13%	14%
TO	8/45	8/52	30/52	30/49	23/49	23/50
	18%	15%	58%	61%	47%	46%
Annuald	9%	8%	19%	20%	16%	15%	15%	15%
NoI	8/43	8/51	2/51	2/30	12/30	12/39
	19%	16%	4%	7%	40%	31%
Annuald	9%	8%	1%	2%	13%	10%	8%	7%
Total	26/134	26/159	61/159	61/129	52/129	52/138
	19%	16%	38%	47%	40%	37%
Annuald	10%	8%	13%	16%	13%	13%	12%	13%

^aThe number and percentage of new caregivers assessed at the second assessment relative to the total number of caregivers assessed at the first assessment.

^bThe number of new caregivers assessed at the second assessment relative to the total number of caregivers assessed at the second assessment.

^cThe average annual rate weighted by the number of years in each of the three time periods.

^dApproximate annual turnover rate for each period of time; 18 months is used as the length of T1 to T2. Annual rate may not equal period rate divided by years because of rounding.

Two different estimates of staff turnover were calculated. The first represented the number and percentage of new caregivers assessed at the second assessment during a time period relative to the total number of caregivers assessed at the first assessment during that time period. This constituted the number and percentage of new caregivers hired over the time period relative to the number at the beginning of the time period. The second estimate of staff turnover represented the number of new caregivers assessed at the second assessment during the designated time period relative to the total number of caregivers assessed at that second assessment. This was equal to the percentage of new caregivers at the second assessment. Both estimates are presented in Table 1, and the percentage turnover as well as the annualized percentage for that time period (boldface) are also presented, the latter being the most comparable figure across time periods and between BHs.

First, the marginal annualized turnover rates for the two estimates were very similar within BHs and within time periods. Second, the weighted average annual turnover rates for T+SC and TO were very similar (14%–16%), but the rate for NoI was somewhat lower (7%–8%).

Third, turnover rates averaged 13% per year and generally were not persistently different for the three BHs, minimizing the potential confounding role of turnover. But rates were somewhat variable across time period and for BHs within time periods, especially 2005–2008 and 2008–2011. Intervention and administrative circumstances and changes in society may have contributed to this variability. The T+SC BH had somewhat higher turnover rates between the end of the intervention and the beginning of follow-up (2005–2008), because the intervention project had supported a few additional staff, some of whom needed to be let go afterwards. The TO BH had higher turnover rates during the last two time periods, because they let go part-time staff in favor of keeping staff who would work more days per week. NoI was threatened by the local government with being closed entirely during the middle time period, but ultimately remained open with approximately half the number of children and caregivers. The slight increase in turnover rates after 2005 may reflect an improving Russian economy that offered more alternative employment opportunities for caregivers.

Maintenance of BH Differences in HOME Inventory Scores

Two types of analyses over the eight-year period of the current study were performed to deal with the fact that caregivers turn over at approximately 13% a year and only 32 of the original 134 (24%) caregivers across the three BHs had all four HOME assessments. The first type of analysis was a quasi-cross-sectional analysis in which the sample of all caregivers available during a given assessment period was used even though some caregivers were represented in more than one time period. These “quasi-independent group analyses” violated the assumption of independence, but that violation would reduce the sensitivity and power of the analysis relative to a longitudinal approach. This analysis provided a “snapshot” of the environment and caregiver behavior of these BHs at each time point, and included the maximum number of caregivers. A second approach was to use overlapping longitudinal segments (e.g., Time 1–2, Time 2–3, Time 3–4), which provided information on true intra-individual change or stability in HOME scores for caregivers who were present at both adjacent assessments, albeit with smaller sample sizes and over shorter spans of time.

A major question was whether the differences between BHs in HOME Inventory scores produced by the intervention conditions would be maintained after the intervention project ended and through the follow-up project. Table 2 presents the means, standard deviations, and Ns for the three Intervention Conditions (T+SC, TO, NoI) at five time points (Pre-Intervention, Intervention Times 1 and 2, and Follow-up Times 3 and 4), and Figure 1 presents the results graphically. Of primary interest in this study are the data from the four time periods following the intervention; the pre-intervention means indicate the status of the three BHs prior to the intervention to illustrate the amount of change subsequently produced by the intervention. During intervention Time 1 and Time 2, only the scores for T+SC were significantly higher than their pre-intervention scores (Table 2). An analysis of variance of Intervention Condition (T+SC, TO, NoI) X Time (1–4) produced a significant difference between the three Intervention Conditions, F(2,548)=84.39, p<.001, eta²=.24, and no BH X Time interaction, F<1. This indicates that the interventions produced differences between the BHs, especially T+SC vs. the other two BHs, and these differences between BHs remained over a period of at least eight years, six years after the intervention project ended.

Table 2.

Mean (SD) HOME Inventory Total Scores From Pre-Intervention Through Follow-up for Three Orphanage Intervention Conditions

		T+SC	TO	NoI
Pre-Intervention		35.1(4.3)	34.5(3.2)	33.3(3.4)
		(N=59)	(N=53)	(N=52)
Intervention	Time 1	37.2(2.4)***	34.7(3.1)	33.4(3.0)
		(N=46)	(N=45)	(N=43)
	Time 2	36.6(3.0)**	34.7(3.1)	32.6(4.1)
		(N=56)	(N=52)	(N=51)
Follow-up	Time 3	39.4(2.2)***	38.0(2.8)***	34.7(3.1)†
		(N=50)	(N=49)	(N=30)
	Time 4	39.6(2.2)***	37.8(2.6)***	35.5(4.2)***
		(N=49)	(N=50)	(N=39)

^†p=.06,

^*p≤.05,

^**p≤.01,

^***p≤.001 for simple effects tests of HOME Inventory Total Score vs. Pre-Intervention level for that BH.

Figure 1.

Mean HOME total score as a function of time for the three intervention conditions

This analysis also produced a significant Time effect, F(3,548)=36.34, p<.001, eta²=.17 (Figure 1), consisting of an increase in total HOME scores between the end of the intervention (T2) and the beginning of the follow-up period (T3). Importantly, however, this occurred for all three BHs and the differences between the three BHs remained. Scheffé tests comparing each pair of time points indicated that Intervention Times 1 and 2 and Follow-up Times 3 and 4 were not significantly different (p>.62) but each of the two Intervention Times were significantly different from each of the two Follow-up Times (p<.001).

Results were similar when the Sociability score and the six conventional subscales of the HOME were analyzed. For the Sociability score, there was a BH effect, F(2,548)=69.44, p<.001, eta²=.20, but no BH X Time interaction, F(6,548)=1.47. A Time effect, F(3,548)=36.99, p<.001, eta²=.17, was accompanied by the same pairwise results as for total score (i.e., Intervention < Follow-up). MANOVA results for the six HOME subscales were similar with a BH effect, F(12,1086)=26.46, p<.001, and a Time effect, F(36,2387)=3.04, p<.001. Although the multivariate interaction was significant in this case, univariate tests indicated a non-systematic pattern of scores for individual BHs over time.

Overlapping longitudinal analyses

The means, standard deviations, Ns, and statistical results for three overlapping longitudinal analyses comparing Time 1–2, Time 2–3, and Time 3–4 are presented at the top of Table 3. The means for these longitudinal samples are very close to the quasi-cross sectional samples (Table 2). The analyses of variance conducted within each of these three time periods were very uniform in their main results, specifically that there was an Intervention Condition effect in each time period accounting for partial eta² of .21–.37, and at the same time in each case the Intervention Condition X Time interaction had an F≤1. A significant Time effect occurred between the end of intervention and the beginning of follow-up (2005–2008) consistent with the quasi-cross-sectional analysis reported above.

Table 3.

Overlapping Longitudinal Analyses of Intervention Conditions Over Years (Top) and Comparison of Old and New Caregivers at Three Time Points for HOME Inventory Total Scores

Longitudinal	2003 (T1) –	2005 (T2)	2005 (T2) – 2008 (T3)		2008 (T3) – 2011 (T4)
	Intervention		Intervention	Follow-up	Follow-up
	T1	T2	T2	T3	T3	T4
T+SC (N=46,21,24)	37.2 (2.4)	36.7(3.0)	37.0 (2.2)	38.8(2.4)	39.3(2.6)	40.2(1.9)
TO(N=44,19,17)	34.8(3.1)	34.6(3.2)	35.2(3.2)	38.0(2.9)	37.4(3.5)	38.1(2.5)
NoI (N=43,28,19)	33.4(3.0)	32.6(4.1)	32.9(3.5)	34.4(3.0)	34.3(3.0)	35.1(3.6)
BH	F(2,260)=34.47***, eta2=.21		F(2,130)=26.41***, eta2=.29		F(2,114)=33.91***, eta2=.37
Time	F(1,260)=1.80		F(1,130)=15.57***		F(1,114)=2.34
BH & Time	F(2,260) ≤ 1		F(2,130) ≤ 1		F(2,114) ≤ 1
Cross-Sectional	T2		T3		T4
	Old	New	Old	New	Old	New
T+SC	36.7(3.0)	36.6(2.9)	38.8(2.4)	39.9(1.9)	40.2 (1.9)	39.0(2.3)
	(N=46)	(N=10)	(N=21)	(N=29)	(N=24)	(N=25)
TO	34.6(3.2)	34.9(2.9)	38.0(2.9)	38.0(2.7)	38.1 (2.5)	37.6(2.6)
	(N=44)	(N=8)	(N=19)	(N=30)	(N=17)	(N=33)
NoI	32.6(4.1)	32.4(4.1)	34.4(3.0)	38.5(0.7)	35.1 (3.6)	36.0(4.8)
	(N=43)	(N=8)	(N=28)	(N=2)	(N=19)	(N=20)
BH	F(2,153)=10.81***, eta2=.12		F(2,123)=5.65**, eta2=.08		F(2,132)=20.15***, eta2=.23
Old vs. New	F(1,153) ≤ 1		F(1,123)=5.76*, eta2=.05		F(1,132) ≤ 1
BH x Old/New	F(2,153) ≤ 1		F(2,123)=2.15		F(2,132)=1.40

Simple effects tests between each pair of Intervention Conditions were conducted within the context of each of these three longitudinal analyses, and the three BHs were also compared separately at Time 2, Time 3, and Time 4. In each of these comparisons, a significant difference was produced between each pair of Intervention Conditions, except in the longitudinal analysis across Times 2 and 3 in which both T+SC and TO were significantly different from NoI but not from each other. It should be noted that by T3 and T4 all three BHs were significantly different from one another indicating that the T3 score for TO was an unusual deviation within that period.

Therefore, both quasi-cross-sectional and overlapping longitudinal analyses show that the intervention-produced differences in HOME scores between the three BHs were maintained throughout the intervention project as well as for six years afterward during the follow-up project.

Old vs. new caregivers

It was of some interest to note whether replacement caregivers (“new”), who received a mini-training course, quickly adapted to the caregiving environment of their particular intervention condition relative to caregivers who had been in that BH for at least 2–3 years. Consequently, separately at Times 2, 3, and 4, caregivers were divided into those who had been present for the previous HOME assessment (“old” caregivers) vs. those for whom the current HOME assessment was their first (“new”).

The means, standard deviations, Ns, and statistical results for these comparisons are presented in the bottom of Table 3. The statistical analysis consisted of Intervention Condition X Old vs. New Caregivers for the HOME assessment at a particular time point. Again, the results were quite uniform in showing that BH differences were highly significant at each of the three time points, and simple effects tests showed that in each case all three BHs were pairwise significantly different from one another. Further, there was no significant Intervention Condition X Old vs. New Caregivers interaction, indicating that new caregivers adjusted to the level of caregiving in their respective BHs. At one time point (2005–2008), there was a significant difference between old and new caregivers, but ironically this was a case in which new caregivers scored higher on the HOME than the older caregivers, but again without interaction with Intervention Condition.

Children’s Battelle Subscale Departure Scores

Table 4 presents the departure Battelle DQ means (SDs) for the personal-social and communication subscales as a function of the three intervention conditions and the three time points. These data were analyzed with an Intervention Condition (T+SC, TO, NoI) X Time (Intervention, Follow-up Time 1, Follow-up Time 2) independent groups MANOVA. Preliminary tests indicated no effects for age at placement, nor did age at placement meet the requirements for it to be used as a covariate in these analyses. The main results of interest were a significant multivariate Intervention Condition effect, F(4, 694) = 7.58, p≤.001, eta² = .04, and no interaction with Time, F(8, 694) = 1.57, p = .13. The Intervention Condition effect was significant in univariate tests for personal-social, F(2, 348 )= 5.50, p = .004, eta² = .03, and for communication, F(2, 348) = 15.45, p≤.001, eta² = .08.

Consistent with the HOME results, there was also a Time effect, F(4, 694) = 3.30, p = .011, eta² = .02, and this was significant in univariate tests for both personal-social, F(2, 348) = 5.74, p = .004, eta² = .03, and communication, F(2, 348) = 3.78, p = .024, eta² = .02. Although there was no significant multivariate Intervention Condition X Time interaction as reported above, the univariate tests of this interaction approached significance for personal-social, F(4, 348) = 2.15, p = .074, and were significant for communication, F(4, 348) = 2.88, p = .023, eta² = .03.

These effects can be seen in Figure 2. The overall multivariate results indicated that the three Intervention Conditions maintained their differences across these three time periods. Scores tended to increase between the Intervention and Follow-up Time 1. The multivariate interaction was not significant. While T+SC maintained much higher scores than NoI throughout the follow-up period, TO’s scores during the follow-up period were as high as those from T+SC.

Figure 2.

Personal-social (solid) and communication (open) Battelle subscale scores for the three intervention conditions as a function of time

Discussion

Main Results

This paper demonstrates that a comprehensive intervention in a St. Petersburg Baby Home was maintained and continued to be associated with better developmental scores for children through the intervention project period as well as for approximately six years after the intervention project ended compared to two comparison BHs. The interventions consisted of T+SC, which included training caregivers in warm, sensitive, contingently-responsive, and child-directed caregiver-child interactions plus a variety of structural changes that reduced group sizes, assigned primary and secondary caregivers, eliminated periodic graduations to wards, integrated groups by age and disability status, and promoted more consistent relationships with fewer caregivers. Another Baby Home (BH) received training only (TO) without the structural changes, and a third had no intervention (NoI).

The results were quite uniform for the HOME Inventory total scores, a sociability index made up of 21 items reflecting caregiver-child social interactions, and the HOME subscales. They demonstrated that the intervention produced an increase in scores in T+SC, but not TO and NoI, from Pre-Intervention levels to Intervention Times 1 and 2 (approximately the last 1–2 years of the intervention project). Moreover, across four time points spanning approximately eight years (Intervention Time 1 and 2, Follow-up Time 3 and 4), the differences between Intervention Conditions (BHs) remained significant and did not interact with time.

In addition, HOME scores remained steady over the two time points during the intervention and the two time points during follow-up, but they rose unexpectedly over the three years between the end of the intervention project and the beginning of the follow-up (Times 2 and 3, 2005–2008). This was true for HOME total score, the sociability index, and the set of six HOME subscales, and it was also true for all three intervention conditions. This increase following the end of the intervention project contrasts with the usual expectation that intervention effects tend to fade over time.

The Battelle Developmental Inventory’s personal-social and communication subscales essentially mirrored the trend in HOME scores. Specifically, the multivariate differences in children’s behavioral development scores produced by Intervention Conditions were maintained during the last years of the intervention project and for six years thereafter with no multivariate interaction with time. Similar to the HOME results, Battelle scores increased after the intervention period; and although the multivariate interaction was not significant, univariate results showed this change over time to increase for TO and NoI but not for T+SC, and TO’s scores approached or exceeded those of T+SC during the follow-up period.

Maintenance of Intervention Effects

Several factors might have contributed to the maintenance of intervention effects on caregivers, the environment, and children’s development.

First, the nature of the intervention may be a factor. In contrast to interventions consisting of supplementary stimulation often provided by research assistants or special supplementary staff in which effects typically diminish after the intervention is terminated, the current intervention was aimed at changing the behavior of all regular staff members in an entire institution. Moreover, the intervention did not focus on having caregivers perform specific actions with children, but rather emphasized the general principles of warm, sensitive, contingently-responsive, child-directed caregiver-child interactions which caregivers could then implement on the wards in a variety of different ways to fit individual children. More speculatively, it is possible that the children themselves helped to maintain the caregiver-child interactions. During the intervention, children’s development in T+SC improved in noticeable ways; children were more active and responsive to caregivers, and they expected caregivers to talk and interact with them. Although this was more visible in older children, age integration was part of the T+SC intervention so that older children resided in every ward. Thus, it is possible that the reciprocal nature of caregiver-child interactions helps to maintain those sensitive and responsive engagements.

A second contributor was the deliberate attempts during the planning of the intervention to help it persist (Altman, 1995). A train-the-trainer strategy had been employed so trainers would be available to train replacement (“new”) caregivers. Such a strategy may have contributed to the finding that new caregivers scored the same on the HOME as old caregivers at each time point within each BH. Also, the fact that BH differences were the same for old and new caregivers suggests that caregivers may also adapt to the caregiving environment in which they work. Further, an intervention that reduced the numbers of caregivers in children’s lives, reduced the number of children per caregiver, but did not increase the total number of caregiver hours could be maintained after the initial grant period using only the government-supplied budget for the BH.

Third, the importance of having a director committed to the intervention should not be minimized. In the authors’ experience in this and studies of interventions in other institutions, a committed director is imperative; without it, staff remain unmotivated to change. It is important to note in the present study that the directors of the other two BHs also wanted their respective treatments, including the Director of NoI who believed in the effectiveness of the existing institutional procedures.

Fourth, certain activities during the follow-up period probably helped. For example, the monitoring and supervision system that was established in T+SC as part of the intervention and continued thereafter was designed to continuously encourage caregivers to implement on the wards the training they had received. Further, two of the authors and two other staff visited T+SC and TO periodically to remind caregivers of their training and provide additional training sessions in the years following the end of the intervention project.

Staff Turnover Rates

Staff turnover has the potential to corrode the maintenance of the intervention. Turnover rates in these BHs averaged approximately 13% per year, which was relatively consistent across the three BHs and across time, with a few exceptions. The NoI BH was threatened with a total shut down, but ultimately was allowed to remain open with approximately half the child population and fewer caregivers. This resulted in extremely low turnover rates between 2005 and 2008. Higher rates were reported in the other two BHs, because the number of children being sent to orphanages declined during this period of time; BHs tried to maintain staff levels, but some caregivers left to take other jobs.

A turnover rate of 13% per year could mean that one in four caregivers is replaced in two years and more than one in three in three years – a span of time that characterizes the residential term of some infants and young children. Indeed, turnover rate plus other factors contributing to many and changing caregivers meant that in these BHs during the intervention years (2003–2005), children experienced 60–100 different caregivers in TO and NoI by 19 months of age (the intervention in T+SC reduced this figure by approximately half; The St. Petersburg-USA Orphanage Research Team, 2008).

At a rate of 13% per year, one might estimate a turnover rate of (8 years x 13%) > 100% over the eight-year period. In fact, however, of the original 134 caregivers assessed at Time 1, 32 (24%) remained at Time 4, which suggests a cumulative eight-year turnover of 76%, not 100%. This implies that a core group of caregivers remain in the BHs over a prolonged period of time, while some of the remaining caregiver positions may turn over several times over the eight years.

The generalizability of these St. Petersburg (RF) staff-turnover rates to other institutions around the world is uncertain. On the one hand, the three BHs were relatively consistent in their turnover rates across the eight-year period. Caregiving in St. Petersburg is a relatively low-status, low-paying job, although contributions are made by the government toward caregiver pensions. This may not be the case in every country. For example, in some Central American countries caregiving in an institution is a government job that brings with it an acceptable (rather than low) salary and fairly permanent government employment that is threatened if a caregiver leaves for a non-government job. In this environment, caregivers may be much more reluctant to leave their institutional employment. This speculation prompts a possible suggestion of increasing the salaries, pension contribution, or job security of caregivers as a policy change that might promote less staff turnover and more consistent caregivers in the lives of institutionalized children.

Societal Improvements

An unexpected finding was that not only were caregiver behaviors and children’s development maintained after the intervention, but they actually improved. It is unlikely that the factors designed to maintain the intervention produced this subsequent increase in HOME and Battelle scores, because the improvement occurred in each BH and only between the end of the intervention and the beginning of the follow-up; maintenance factors existed within the intervention period and within the follow-up period, but no such improvement occurred during these intervals.

We can only speculate about events that occurred in St. Petersburg predominantly between 2005 and 2008 that might have influenced caregiver behavior and the caregiving environment and presumably its effects on children’s development (Rosas, McCall, Groark, Muhamedrahimov, Palmov, & Nikiforova, 2011). Mainly, the government began special efforts to keep children in families and reduce the number of children being sent to BHs. For example, Social Services began to remove fewer infants and toddlers from at-risk families and provided at-risk families, single mothers, and mothers with special needs with more financial help as well as food and clothing. Also, a media campaign encouraged family care alternatives, and the Russian economy improved; as a result there were fewer “social risk” children, some of whom previously had been relinquished to BHs, and more families could afford to adopt.

While the number of children being sent to BHs in St. Petersburg declined substantially (e.g., BH populations declined approximately 10–15%), BH directors attempted to maintain their staffs. The net result was smaller groups of children cared for by nearly the same number of caregivers, which produced lower children:caregiver ratios. Thus, the number of different caregivers experienced by a child likely did not change greatly during the intervention in T+SC but this factor was not found to mediate intervention effects (Rosas et al., 2011); however the number of children per caregiver did decrease. It is possible that when confronted with fewer children, caregivers are less rushed and have more time to interact with children, and this environment draws more typical adult-child behavior from the caregivers. Thus, smaller children:caregiver ratios may be a major contributor to institutionalized children’s development, a proposition consistent with differences between institutions in this regard and children’s development (Groark, McCall, Fish, McCarthy, Eichner, & Gee, 2012) and the literature on non-residential child care (e.g., Kontos & Fiene, 1987; NICHD Early Childcare Research Network, 2000).

Even if this explanation were correct, it is surprising that smaller groups and fewer children per caregiver alone could apparently produce as much or more improvement in HOME scores as the original comprehensive intervention. But dramatic improvement over time in HOME scales may be associated with the HOME scale itself and its application to group settings. Items on the HOME are scored “yes” or “no” as a function of whether the caregiver performed the indicated action at least once with at least one child during the one-hour observation period. With fewer children and more time available for interactions with them, it may not be surprising that caregivers had more opportunity to perform the behaviors on the HOME at least once during the observation period.

Intervention differences in children’s personal-social and communication development were also maintained during and after the intervention, and these scores also tended to improve between the end of intervention and follow-up but not as dramatically as the HOME scale. The intervention in T+SC produced marked changes in total Battelle Inventory scores, improving children exposed to the T+SC intervention for nine months or more from an average of DQ=57 to 92 = 35 DQ points (The St. Petersburg-USA Orphanage Research Team, 2008). Thus, we believe that reducing group size and the number of children per caregiver likely improved caregiver behavior with children and children’s development in otherwise deficient environments, but its effects were not as strong as those produced by a comprehensive intervention.

The most unexpected finding was the substantial apparent improvement in Battelle scores for TO children (Figure 2) between the intervention and follow-up periods to levels equal to or exceeding those for T+SC children (although the interaction was not statistically significant). One possible explanation is that TO was encouraged to implement the same structural changes as T+SC during this time period; although this was only partly accomplished, it may have produced some or all of this improvement.

A second explanation pertains to changes in placement practices, partly in response to the decline in the number of children being sent to BHs generally. The TO BH tried to keep the number of children in residence as high as possible so they could justify retaining staff. This BH is located next to a maternity hospital, which enable the BH to take many of their children whose legal and developmental status were such that they could be more quickly transitioned to domestic families. These infants had significantly higher birth weights, lengths, and head circumferences than infants distributed by the health administration to the other BHs. These children tended to be transferred to adoptive and foster families, but the total number of children transitioned nevertheless declined in part because some of the transitions occurred before the 3-month residency requirement of this study. The NoI BH was reduced by the administration to approximately half its original size. In contrast, T+SC had always been willing to take children with more severe health and developmental problems and disabilities; such children became a higher percentage of those being sent to BHs, and the T+SC Director was vigorous in trying to reunify many of these children with their biological families (i.e. 40% of the T+SC vs. 15% of the TO children). These recent historical changes meant that the number of children transitioning to families declined in TO and NoI but not in T+SC, and the developmental level of such children (Fig. 2) appears to have increased somewhat in TO and declined a bit in T+SC (although this interaction was not statistically significant).

Assets and Limitations

The St. Petersburg study is the most comprehensive intervention in institutions for infants and toddlers using regular institutional staff caregivers (Bakermans-Kranenburg, van IJzendoorn, & Juffer, 2008), and it is the most comprehensively evaluated institutional intervention. It is also the only one that made deliberate attempts in the planning of the intervention as well as after the intervention project terminated to support the maintenance of the intervention and its benefits for children. Further, this is the only report of a successful attempt to maintain an intervention in an institution for infants and toddlers over the years of the intervention project itself as well as for six years after its termination.

This report has several limitations. Unfortunately, funding was not available for the two years between the end of the intervention and the start of the follow-up project, and thus no children departing these institutions during this two-year interval were given departure assessments. Further, only two Battelle subscales were administered to children at departure during the follow-up compared to the full-scale Battelle and other behavioral measures administered during the intervention project. Moreover, several events beyond our control occurred after the intervention terminated. For example, the NoI BH was reduced to approximately half of its original size. Further, changes in St. Petersburg and the Russian Federation reduced the number of children being sent to BH’s but the number of staff declined less, producing smaller group sizes and fewer children per caregiver, which we interpret to have produced higher HOME and Battelle scores, especially in TO and NoI. Further, the local government changed how they placed children in these institutions and that may have influenced children’s departure scores.

Is it possible for the societal changes to explain the main finding of the maintenance of intervention effects? Although such factors likely contributed to the increase in HOME and Battelle scores over time, the general result was that the differences between the intervention conditions were maintained across time, which could not easily be explained by societal changes that applied to all three BH’s. Moreover, although TO appeared to catch up with T+SC with respect to children’s development but not HOME scores, this appears tied to the government sending TO more developmentally advanced children, because the difference between T+SC and NoI remained substantial.

A major limitation would appear to be the lack in this report of pre-intervention data to demonstrate that the differences between BHs that were maintained during and following the intervention were not associated with pre-existing developmental differences in the children. The report of the original intervention (The St. Petersburg-USA Orphanage Research Team, 2008) plots and analyzes baseline-to-departure changes or covaries initial status, making it quite clear that the intervention effects---that were maintained over six years in the current study---were not due to selective sampling differences between BHs. The same is true for the maintenance of the HOME results.

A related concern is the possibility that BHs were assigned different types of children over the years and that children destined for different types of families were developmentally different. A recently completed study of the same children represented in this report shows that children who were transitioned to USA adoptive families and St. Petersburg adoptive, foster, and biological families from these three BHs were not different in birth weight, length, and head circumference; departure height, weight, and Battelle scores; and length of residence in the BH. Thus, although some shifting in allocations and departure destinations between BHs did occur during follow-up, the lack of overall developmental differences associated with these factors minimizes the likelihood that they played a major role in the maintenance effect.

Another limitation is that the current study used only those BH children who transitioned to USA families through one adoption agency and to Russian families. It seems unlikely that the reported results are unique to this subsample. The children studied are indeed selected (by parents), but by far the largest group in the BHs who were not studied are also selected for adoption to the USA and Western Europe but through other agencies. Agreed, children who remain in the BHs are more likely to have disabilities and would score lower, but T+SC would have a disproportionate number of these. Even so, the T+SC intervention also improved the development of children with disabilities (The St. Petersburg-USA Orphanage Research Team, 2008), and there is no reason to believe that the maintained environment in T+SC would not continue to improve children with disabilities and others who remained in the BH.

Finally, the literature reviewed above emphasizes that intervention effects are more likely to be maintained if the intervention itself is created in a way to support maintenance and if various activities aimed at maintenance are conducted after the intervention project is terminated, and we believe the steps taken in these regards contributed to the maintenance we observed. Nevertheless, the current study does not provide direct support for this speculation, because no control condition was conducted that omitted those activities. Further, this study only reports the “maintenance” of an intervention after the intervention project terminated in the same institutions and with some of the original researchers still involved with these institutions. Thus, this is only the first step in the broader process of “sustainability,” which involves transferring the project to new institutions and new staff.