Factors That Influence the Effectiveness of Child Protection Teams

Abstract

OBJECTIVES

More than $55 million is spent on hospital-based child protection teams (CPTs) annually, but there is no consensus on what makes CPTs effective. The objective of this study was to create expert consensus on tasks that CPTs should perform and factors that contribute to effectiveness.

METHODS

A modified Delphi approach was used to create expert consensus among professionals with experience working on or with hospital-based CPTs. Three initial rounds of surveys were conducted; a first round of open-ended questions generated topics related to CPT tasks and factors related to team effectiveness. A Likert scale (range: 1–7) determined rank. In the fourth round, participants ranked the top 5 variables associated with effectiveness.

RESULTS

Twenty-six (90%) of 29 participants completed the first 3 rounds, and 20 (67%) completed the final ranking. Experts believed that CPTs should provide communication of findings to appropriate agencies (mean Likert score: 7.0), court testimony (7.0), medical consultations (6.9), multidisciplinary case review (6.6), and forensic interviews (6.0). CPT success should be determined by professionals who use CPT services (6.6) and CPT members (6.5). Variables that were ranked most often as critical to effectiveness included interdisciplinary collaboration (95% of participants), provision of resources (80%), and team collegiality (75%). Variables that were ranked as most detrimental included inadequate staffing (85%) and lack of collegiality (80%).

CONCLUSIONS

A multidisciplinary team working in a collegial atmosphere seems to be the major key to CPT effectiveness. In addition to providing services, CPTs should focus on improving collegiality and interdisciplinary collaboration and should seek performance feedback from referring professionals and CPT members.

Each year, many children sustain serious injuries and death as a result of abuse. In 2007, 3.2 million reports concerning suspected maltreatment of ~5.8 million children were made to child protective services (CPS) agencies across the United States.¹ Of these, ~794 000 children were found to be victims of child maltreatment, and an estimated 1760 children died as a result of abuse or neglect.¹ CPS agencies are often overwhelmed with cases because of a combination of underfunding and high demand, which can lead to allocation of scarce resources to the investigation of severe cases of child abuse and neglect (CAN), leaving fewer resources available for other potentially significant cases.^2,3

To help address the challenges of CAN, hospital-based child protection teams (CPTs) were proposed >50 years ago as an approach to the care of abused children.^4–6 These multidisciplinary teams of child abuse experts share a common goal of maximizing the quality and coordination of care of the maltreated child, with the aim of improving child health outcomes. A number of studies have described the compositions and goals of various hospital-based CPTs and have shown that the teams vary widely in both structure and function.^6–18 No systematic evaluation of the effectiveness of CPTs has been performed, in large part because of the variation in team composition and lack of a formal consensus as to what the tasks of CPTs should be or how team function should be measured.

To evaluate and compare CPTs, we applied the theories of Hackman’s model of group effectiveness.¹⁹ This model is a well-accepted framework used to evaluate team function and has been applied to a variety of case studies, including health care teams.^19,20 Hackman’s model includes 3 outcomes: (1) the degree to which the group’s productive output meets the standards of quantity, quality, and timeliness of the people who receive, review, or use that output; (2) the degree to which the process of carrying out the work enhances the capability of team members to work together again; and (3) the degree to which the group experience contributes to the growth and personal well-being of team members. To determine what makes a CPT effective, we adapted these theories to address each of these outcomes and developed a modified Delphi approach with the goal of establishing expert consensus on the tasks that CPTs should perform; the way in which CPT performance should be evaluated; and the team structures, group dynamics, and organizational contexts that contribute to team effectiveness.

METHODS

The Delphi technique has been used to achieve consensus in situations in which unanimity of opinion does not exist.^21–23 It is an iterative process that involves a series of rounds of data collection, typically by using anonymous questionnaires.^22,23 In this study, a modified Delphi method consisting of 3 rounds of self-administered surveys was used to establish expert consensus among 29 professionals from 16 states who consulted with or were members of a CPT or who otherwise had CAN professional experience. The first round used open-ended questions to generate a wide range of topics related to tasks that CPTs should perform, the evaluation of team performance, characteristics of hospitals that contribute to higher quality CPTs, and factors related to hospital-based CPT group effectiveness. In the second round, the participants used a Likert scale from 1 to 7 to rank their level of agreement with statements generated from round 1. In the third round, participants had the opportunity to compare their responses with the group average and adjust their responses. The statements with a mean Likert score of ≥6 in round 3 were considered to have reached consensus. Finally, participants were asked to rank the tasks and characteristics that they believed were most important of the statements that achieved consensus.

Participant Selection

The participants were identified and recruited by surveying a national sample of CAN professionals and pediatricians. The Child Maltreatment Conference, the largest annual multidisciplinary child abuse conference, is attended by ~2000 participants who are representative of professionals who are from multiple disciplines and geographic locations and work in the area of child abuse. A survey was mailed to participants of the 2005 Child Maltreatment Conference requesting the nomination of experts who are child abuse physician specialists, social workers, and CPS workers with experience working with hospital-based CPTs. The individuals named most frequently from each discipline were identified and invited to participate. In addition, an invitation was sent to members of the Academic Pediatric Association (formally known as the Ambulatory Pediatric Association) to solicit the participation of primary care pediatricians in the study. Individuals who were nominated or who volunteered to participate were screened to ensure that they met criteria for classification into 1 of 4 professional groups: primary care pediatricians, child abuse physician specialists, social workers, or CPS workers. Potential participants were also screened to ensure that they were either members of a hospital-based CPT or had referred patients to their local hospital-based CPT.

Delphi Process

Round 1

In the first round, the participants completed demographic data forms and were asked a series of open-ended questions about the tasks that CPTs should perform, the evaluation of CPT performance, and factors related to CPT effectiveness (Table 1). The questions for this round were based on the theories of Hackman’s model of group effectiveness.

TABLE 1.

Delphi Round 1 Questionnaire

Which specific tasks are important for a hospital-based CPT to perform?

How should we measure how successful a CPT is in performing its tasks?

Who should measure how successful a CPT is in performing its tasks?

Which characteristics of how a CPT works together do you think enhance the ability of team members to work together again?

Which characteristics do you think are detrimental?

Which characteristics of an institution do you think enhance the capability of the members of a CPT to work together again?

Which characteristics do you think are detrimental?

Which parts of the experience of participating as a member of a CPT do you think contribute to each member’s professional growth?

Which parts do you think are detrimental to a member’s professional growth?

Which components of the experience of participating as a member of a CPT do you think contribute to each member’s sense of well-being on the team?

Which components do you think are detrimental to a member’s sense of well-being?

Which 3 characteristics do you think distinguish high-quality CPTs from other CPTs?

After completing round 1, the participants’ responses were collected and transcribed into Qualitative Solutions Research NVivo 7 software, in which they were coded into categories of determinants of group effectiveness. NVivo subcategories were generated to create “node addresses” for each of the text units, which allowed for tracking of the number and percentage of responses related to each subcategory. The research team validated the initial coding by reviewing a sample of the transcript and codes that were assigned to each statement. Disagreements in coding were resolved by discussion with members of the research team (I.T. and H.B.) until full agreement was reached. Codes that were generated from round 1 were turned into statements about each of the structural and dynamic issues surrounding CPT success. The statements were then written into a survey format and used in round 2.

Round 2

In round 2, participants reviewed the statements that were generated from round 1 and ranked them on a Likert scale from least important (1) to most important (7). Space was provided for written responses in case clarification or discussion was desired. Numerical rankings were entered into SAS software (SAS Institute, Cary, NC) to perform measurements of central tendency. A statement was retained for the subsequent survey round when 75% of participants ranked it ≥6 on the 7-point Likert scale.

Round 3

During round 3, participants were given the distribution of responses to each statement in round 2 as well as the relationship of their own responses to the responses of the rest of the group. Next to each statement, both the group mean rank and the individual’s own rank of the importance of the statement from round 2 were displayed. Participants were asked to reevaluate each statement and were given the opportunity either to retain their own rank or to change it. Space was provided for comments, clarification, or discussion on each statement. This round was analyzed similarly to round 2. Statements with an average Likert response of ≥6 were considered to have reached consensus.

Round 4: Final Ranking

Of the statements that achieved an average Likert score of ≥6 by the end of round 3, participants were then asked to rank the 5 most important tasks that CPTs should perform, as well as the top variables that they believed were associated with positive effects on CPT function and the top variables that they believed to be detrimental to CPT effectiveness. This was done to identify the most important factors related to CPT effectiveness. Because a small number of statements related to the evaluation of CPT performance reached consensus at the end of round 3, these statements were not included in round 4 and participants were not asked to rank them further.

RESULTS

Of 29 participants, 26 (90%) completed the first 3 rounds of the Delphi process, and 20 (67%) went on to complete the fourth round of ranking the most important tasks and variables related to CPT success. Table 2 summarizes the demographic characteristics of the participants. The expert group was predominantly female (72.4%) and was composed of 7 CAN physicians (24.1%), 6 primary care physicians (20.7%), 8 social workers (27.6%), and 8 child protective service workers (27.6). The majority of the experts had been in their field for at least 5 years (89.6%).

TABLE 2.

Demographics of CAN Experts Who Participated in the Delphi Survey (N = 29)

Demographic	%
Gender
Male	27.6
Female	72.4
Professional discipline
Child abuse physician	24.1
Primary care pediatrician	20.7
Hospital-based social worker	27.6
CPS worker	27.6
Years in current discipline
< 5	10.3
5–10	24.1
11–15	17.2
> 15	48.3
Age, y
30–39	24.1
40–49	44.8
>49	31.0
Region
Northeast (NY, MA)	34.5
Midwest (IA, MN, MO, OH, OK, WI)	24.1
South/Southwest (FL, NM, TX, VA)	17.2
West Coast/West (AK, CA, CO, WA)	24.1
Experience with CPT
CPT medical director	13.8
Co-director of CPT	3.4
Member of CPT	31.0
Former member of CPT	6.9
Member of other team that works with CPT	6.9
Professional who refers cases to CPT	37.9

In round 1, open-ended questions generated 21 statements on tasks that CPTs should perform, 15 statements related to performance evaluation, 45 statements related to variables that are associated with increased CPT effectiveness, and 30 statements related to variables that are detrimental to CPT effectiveness. The number of statements that subsequently reached consensus by the end of rounds 2 through 4 is summarized in Table 3. At the end of round 3, a high number of statements related to CPT tasks and characteristics related to team effectiveness had reached consensus. The statements in these categories were narrowed down in round 4, when participants were asked to rank the tasks and team characteristics that they believed were most significant of all of the statements that reached consensus. Only 5 statements related to CPT performance reached consensus at the end of round 3, so these statements were not included in round 4.

TABLE 3.

Number of Statements That Reached Consensus by the End of Delphi Rounds

Content Area	Round 2	Round 3	Round 4
Tasks that CPTs should perform	17	17	5
Evaluation of CPT performance	13	5	Not needed
Variables associated with CPT effectiveness	44	44	14
Variables that are detrimental to CPT effectiveness	26	27	10

The top 5 tasks that experts believed that CPTs must perform are summarized in Table 4. The tasks were divided into categories of consultation, interdisciplinary work, and participation in legal proceedings. The specific tasks that ranked the highest included providing CAN medical consultations (6.9), facilitating accurate communication of CAN-related findings to appropriate agencies (7.0), participating in multidisciplinary reviews of cases of possible abuse or neglect (6.6), conducting forensic interviews of children who may have been abused or neglected (6.0), and testifying in court (7.0). None of the expert participants ranked provision of medical care to foster children, participation in the development of child abuse prevention projects, or preparation of other professionals for court testimony among the 5 most important tasks that CPTs should perform.

TABLE 4.

Top 5 Tasks That CPTs Should Perform

Team Task Category	Team Task	Mean Likert Response	% Who Ranked Task Among Top 5 Most Important
Consultation	Medical CAN consultations	6.9	85
Interdisciplinary work	Communication of CAN findings to appropriate agencies	7.0	50
	Multidisciplinary review of cases	6.6	60
	Forensic interviews (observed by police, CPS)	6.0	55
Participation in legal proceedings	Testimony	7.0	55

Criteria related to the evaluation of CPT performance are summarized in Table 5. Experts believed that CPT performance should be measured by evaluating whether the involvement of the CPT resulted in more timely investigation of cases of possible CAN (6.6), the provision of more services to families (6.4), and better CAN education of medical professionals who consult with CPTs (6.1). Professional groups who use the services of the CPTs (6.6) and members of CPTs (6.5) should determine whether a CPT is successful in completing its tasks.

TABLE 5.

Evaluation of CPT Performance

Variable Category	Variable	Mean Likert Response
Measure of CPT success at performing its tasks	Involvement of CPT results in more timely investigation into suspected cases of CAN	6.6
	Involvement of CPT results in provision of more services to children and families as well as more timely connection to services and longer term provision of services	6.4
	Professionals who work with CPTs demonstrate increased knowledge of issues related to CAN compared with professionals who do not work with CPTs	6.1
Who should determine whether a CPT is successful at completing its tasks?	Professional groups that use the services of the CPT, including primary care physicians, CPS workers, and police	6.6
	Members of the CPT	6.5

Only 5 statements related to the evaluation of CPT performance reached consensus at the end of round 3, so these statements were not included in round 4 and participants were not asked to rank them further.

Table 6 summarizes the variables associated with CPT effectiveness, divided into categories of hospital variables, variables that lead members to want to work together again, opportunities for professional growth, contributions to member well-being, and characteristics of high-quality CPTs. Many of the variables had an average Likert score ≥6 but differed in the number of participants who ranked them as most important for CPT effectiveness. The variables that were ranked as most important by the highest number of participants were active interdisciplinary collaboration (95%); provision of adequate support staff, space, and equipment by the hospital (80%); a sense of team collegiality (75%); mutual trust and respect (75%); and the opportunity for CAN education (75%). A high perpetrator conviction rate (0%), team longevity (0%), and the perception by outsiders of team cohesiveness (0%) were not believed by any of the participants to be 1 of the top indicators of a high-quality CPT.

TABLE 6.

Variables Associated With CPT Success

Variable Category	Variable	Mean Likert Response	% Who Ranked Variable Among Most Important
Hospital variables	Hospital provides adequate support staff, space, and equipment	6.8	80
	Protected time for team members	6.8	55
	Hospital demonstrates that it supports the activities of the CPT (eg, willing to collaborate with community agencies, reform hospital systems to improve CAN care)	6.8	55
	Adequate financial support	6.7	70
Variables that will lead to CPT members’ working together again	Good communication among team members	7.0	60
	Strong CPT leadership	7.0	50
	Team members feel a sense of collegiality	6.9	75
How participation in CPT would provide opportunity for professional growth	Active interdisciplinary collaboration (eg, members from different disciplines discuss cases, share ideas and experiences, and participate together in critical analysis of cases and interpretation of data)	6.8	95
	Opportunity for CAN education	6.8	75
How CPT contributes to members’ well-being	Feeling that working as a team leads to better outcomes	6.8	75
	Mutual support from members	6.7	70
Characteristics of high-quality CPTs	Members feel mutual trust and respect	6.9	75
	Participants are professionals from multiple disciplines	6.9	55
	CPTs review their own program on a regular basis	6.4	55

Table 7 summarizes the variables that were believed by the participants to be most detrimental to CPT effectiveness, divided into similar categories of hospital variables, variables that deter members from working together again, conditions that are detrimental to professional growth, conditions that are detrimental to member well-being, and characteristics of low-quality CPTs. Inadequate staffing (85%), lack of team collaboration (80%), poor member participation (75%), and perceived lack of respect by the hospital for the CPT (70%) were believed to have a significant negative effect on CPT effectiveness.

TABLE 7.

Variables That Are Believed to Be Detrimental to CPT Success

Variable Category	Variable	Mean Likert Score	% Who Ranked Variable Among Most Important
Hospital variables	Inadequate staffing of the CPT	6.5	85
	Hospital does not demonstrate support or respect for the CPT (eg, is not willing to collaborate with community agencies or reform hospital systems to improve CAN care, does not provide protected time or space for CPT members)	6.4	70
Variables that will deter CPT members from working together again	Members do not work with other team members	7.0	70
	Poor team leadership	6.8	65
How participation in CPT is detrimental to professional growth	Team does not have a collaborative atmosphere	6.9	80
	Team members are not engaged	6.9	75
How CPT is detrimental to members’ well-being	No sense of team accountability for bad outcomes	6.8	70
	Members feel isolated or excluded from the rest of the team	6.7	70
Characteristics of low-quality CPTs	Poor communication among team members	6.9	70
	Team leadership is not responsive to members’ concerns	6. 8	55

DISCUSSION

To the best of our knowledge, this is the first study to establish expert consensus on factors related to CPT performance and team effectiveness. By using the Delphi process, experts identified the key tasks that CPTs should perform and agreed that feedback on CPT performance should be elicited from members of the professional groups who work with CPTs and from CPT members. In addition, experts concluded that hospitals and CPTs need to create an active partnership to produce an effective team. Programs that are interested in improving CPT effectiveness should invest in promoting member collegiality and increasing active interdisciplinary collaboration.

Of the tasks that CPTs should perform that reached consensus in this process, providing medical CAN consultations achieved a high mean Likert response and was the most frequently identified important work. This finding is consistent with the results of a study that examined the system of care in children’s hospitals for victims of CAN that identified consultations as the most common function of CPTs.¹⁸ The same study identified communication with CPS as another common task,¹⁸ which is in keeping with the consensus in our study that communication of CAN findings to appropriate agencies is 1 of the top 5 tasks that CPTs should perform. Forensic interviews, multidisciplinary review of cases, and testimony have not previously been described as some of the most important tasks of CPTs. Future studies are needed to evaluate whether these tasks differ from what is expected of the CPTs by the hospitals.

In Hackman’s model of group effectiveness, teams are evaluated on the production of a high-quality product or service, the willingness of team members to continue to work together, and the team’s contribution to each member’s personal and professional growth. Overall team effectiveness depends on what the team accomplishes and on the manner in which it functions. In our study, experts agreed that CPT performance should be evaluated both by professionals who use CPTs and by CPT members themselves. The timeliness of investigation into suspected cases of CAN, the quality and quantity of services provided to families, and the CAN education of professionals who work with CPTs should be considered when evaluating the quality of services that a CPT provides.

Experts identified hospital support, active interdisciplinary collaboration, and team collegiality as key factors for CPT effectiveness, whereas poor staffing and lack of collaboration within a team were believed to be most detrimental. These findings are similar to those described by Vinokur-Kaplan,²⁰ who used Hackman’s model to evaluate the effectiveness of mental health teams and concluded that it is critical to include environmental, organizational, and interpersonal factors in measures that are used to evaluate and improve team function.

The findings from this study can be used to evaluate hospital-based CPTs in a uniform manner on the variables that were found to be crucial for CPT effectiveness. The National Association of Children’s Hospitals and Related Institutions has published guidelines for developing child protection programs that focus on team structure, staffing, and function.²⁴ The importance of group dynamics and team collegiality, which are not discussed in the guidelines but were highlighted by the respondents in our study, should be taken into account as well. As hospital-based CPTs continue to grow and expand their services, often requiring substantial funding from hospitals,²⁵ it becomes increasingly important to evaluate team effectiveness. CPTs must perform certain tasks related to child protection and should also examine their composition and team processes to maximize their overall success. Additional factors that enable teams to function cohesively in the potentially stressful environment related to decision-making concerning CAN should be evaluated in future studies.

There are 4 limitations to this study, 2 of which relate to the Delphi process. The success of the Delphi process depends on the composition and interest of the group of experts involved. In this study, the participants all had experience working on or with a hospital-based CPT and were knowledgeable about the team’s tasks and functions. Despite this, the group may not have been representative of all of the professionals who work with hospital-based CPTs, and additional perspectives on what makes CPTs successful may be missing. Another potential weakness of the Delphi process is the absence of the opportunity for discussion or elaboration among participants because the process is conducted anonymously. Anonymity, however, may allow participants to express their opinions more freely and engage in the process without influence from strong personalities. Third, in this study, we focused solely on hospital-based CPTs. The factors related to the effectiveness of community-based CPTs may be different from those that we identified. Finally, we did not examine the overall impact of CPTs but rather what experts believed were important CPT functions and which factors make CPTs more effective. The expert panel did identify measures that can be used in the future to evaluate overall team success and examine the relationship between CPT structure and function and child health outcomes. Future studies are needed to examine whether the involvement of CPTs results in better quality and coordination of care for abused children.

CONCLUSIONS

By using a modified Delphi method, we were able to achieve consensus on tasks that CPTs should perform and determine variables that are both beneficial and detrimental to the effectiveness of CPTs, with the belief that improved effectiveness will lead to better outcomes for abused children. A multidisciplinary team working in a collegial atmosphere with active hospital support seems to be the major key to effectiveness. CPTs should focus their internal efforts on improving member collegiality and encouraging active interdisciplinary collaboration and should seek performance feedback from professional groups and CPT members.

WHAT’S KNOWN ON THIS SUBJECT

Each year, more than $55 million is spent on hospital-based CPTs, but there is no consensus on what makes these teams effective or how the teams should be evaluated.

WHAT THIS STUDY ADDS

A multidisciplinary team working in a collegial atmosphere seems to be the major key to CPT effectiveness. In addition to providing services, CPTs should focus on improving collegiality and interdisciplinary collaboration and should seek performance feedback from referring professionals and CPT members.

ABBREVIATIONS

CPS

child protective services

CAN

child abuse and neglect

CPT

child protection team