A review of recent analyses of the Canadian Incidence Study of Reported Child Abuse and Neglect (CIS)

Abstract

Introduction:

The objective of this analysis is to identify, assess the quality and summarize the findings of peer-reviewed articles that used data from the Canadian Incidence Study of Reported Child Abuse and Neglect (CIS) published since November 2011 and data from provincial oversamples of the CIS as well as to illustrate evolving uses of these datasets.

Methods:

Articles were identified from the Public Health Agency of Canada’s data request records tracking access to CIS data and publications produced from that data. At least two raters independently reviewed and appraised the quality of each article.

Results:

A total of 32 articles were included. Common strengths of articles included clearly stated research aims, appropriate control variables and analyses, sufficient sample sizes, appropriate conclusions and relevance to practice or policy. Common problem areas of articles included unclear definitions for variables and inclusion criteria of cases. Articles frequently measured the associations between maltreatment, child, caregiver, household and agency/referral characteristics and investigative outcomes such as opening cases for ongoing services and placement.

Conclusion:

Articles using CIS data were rated positively on most quality indicators. Researchers have recently focussed on inadequately studied categories of maltreatment (exposure to intimate partner violence [IPV]), neglect and emotional maltreatment) and examined factors specific to First Nations children. Data from the CIS oversamples have been underutilized. The use of multivariate analysis techniques has increased.

Key findings

• In this review of 32 peer-reviewed published articles, the majority were of high quality with clearly stated research aims, appropriate control variables, appropriate analyses, sufficient sample sizes, appropriate conclusions and relevance to practice or policy.

• Researchers using CIS data have recently focussed on inadequately studied categories of maltreatment, including exposure to intimate partner violence, neglect and emotional maltreatment, and have examined factors specific to First Nations children.

• The use of complex multivariate analysis methods has recently increased.

Introduction

The Canadian Incidence Study of Reported Child Abuse and Neglect (CIS) has been completed for three cycles—1998,¹ 2003,² and 2008³—and has provided a wealth of information on investigations of reported child abuse and neglect nationwide.⁴ The CIS collects data about children who have been reported to child welfare agencies because of allegations of maltreatment.^1-3 CIS data can be used as a source of information about children who have experienced alleged and substantiated neglect, exposure to intimate partner violence (IPV), emotional (or psychological) maltreatment, physical abuse and/or sexual abuse. It includes information on the characteristics of the child, caregiver, household and investigating agency as well as short-term service outcomes such as placement. CIS data is used by senior child welfare decision makers to help determine resource allocation, identify at-risk populations, understand reported maltreatment trends, validate findings at individual agencies and direct changes in practice.⁴ It is also accessed by a broad range of experts in clinical medicine, public health, social work, law and justice, education, sports, recreation, and faith-based groups.⁵

The CIS–2008: Major Findings report³ details the breadth and scope of the CIS. Each cycle of the CIS contains information on a large sample of maltreatment cases reported to child welfare agencies and opened for investigation. Child welfare workers in selected agencies across Canada completed the CIS survey for each investigation they completed within a three-month data collection period in the fall of the survey year. The participating child welfare workers were given definitions of maltreatment to ensure consistency. A multistage stratified clustered sample from all provinces and territories that included mainstream and First Nations agencies was used. Some provinces and territories provided additional funding to the CIS to obtain data (i.e. oversample) from their respective jurisdictions.

The CIS–2008 study differs from the CIS–1998 and 2003 waves because investigations of risk of future maltreatment were tracked separately from investigations of allegations of specific maltreatment. In addition, definitions of maltreatment have evolved over time. For example, the CIS–2008 tracked three subtypes of exposure to IPV: indirect exposure to physical violence, direct exposure to physical violence and exposure to emotional violence.⁶ Previously, the CIS captured IPV as a subtype of emotional maltreatment. Differences between provincial and territorial legislation and changes in detection, reporting and investigation procedures over time also influence the CIS findings. This means that changes in estimates may not represent actual changes in the occurrence of maltreatment.

An earlier review⁷ summarized and critically assessed 37 peer-reviewed analyses of CIS data published before November 2011. Roughly half of the sources were descriptive and half were multivariate. The review assessed the quality of the articles and found the multivariate articles to be generally of good quality; stronger evidence was provided for their objectives compared to the descriptive articles. The review recommended that future research use clear directional hypotheses and multivariate techniques.⁷

Considering the cyclical nature of the CIS, and especially because the Public Health Agency of Canada decided not to collect data in 2013, this is an opportune time to update the findings from the previous review and ask questions about the relevance of the CIS in terms of policy and practice. Only 3 of the articles in the previous review⁷ utilized data from CIS–2008. In addition, the information was limited because the data had only recently been released. Furthermore, the previous review⁷ was critiqued for not including results stemming from oversampling provinces. As more articles utilizing these and earlier waves of data have since been published, our review focusses on CIS-related literature published after November 2011 and analyses of CIS oversample data published at any time.

Knowledge users (decision makers and policy makers) find effectively summarized research useful. The surveillance reports stemming from the CIS have been used as a reference to obtain information quickly.⁴ Similarly, we hope that this review will provide a quick reference for topics that have been analysed using the CIS data as well as on the quality of these articles. This review may also inspire researchers to further knowledge about child maltreatment and the responses of child welfare agencies. The information may also serve to improve the CIS as a surveillance tool by identifying some gaps in data collection and analysis. Lastly, the information has the potential to increase awareness of this important public health issue.

The specific objectives of this review were to

• identify and retrieve all peer-reviewed studies published between November 2011 and the present day that used CIS data or CIS provincial and territorial oversampling data collected since the inception of the study;

• assess the quality of those studies;

• summarize the findings of those studies; and

• illustrate the evolving uses of CIS data.

Methods

We included original research published in peer-reviewed journals as these can be expected to be of the highest quality and include sufficient information on methods and analyses to assess quality. Other sources, such as book chapters and presentations, may not follow a standardized format of presenting information or have specific objectives or hypotheses to test. As such, our quality assessment tool would not be suited to them. In addition, because the review is restricted to peer-reviewed journal articles, interested readers should find it easier to locate and access our primary sources. Table 1 lists all survey waves of the CIS, jurisdictions that oversampled and First Nations samples.

TABLE 1. CIS, CIS oversamples and First Nations studies.

Population	Cycles
Canada	1998, 2003, 2008
British Columbia	1998, 2008
Alberta	2003, 2008, 2013
Saskatchewan	2008
Ontario	1993, 1998, 2003, 2008, 2013
Quebec	1998, 2008, 2014
Northwest Territories	2003
First Nations	1998, 2003, 2008

Articles were identified through the Public Health Agency of Canada data request records. In addition, we conducted a search for articles published by authors from the CIS research team to ensure completeness. As members of the research team did not need to request permission to use CIS data, their publications would not be captured in data request records.

Figure 1 summarizes the article selection strategy (source, inclusion and exclusion criteria) and presents the list of quality assessment questions. The quality assessment questions were the same as those used in the previous review of CIS data⁷ (discussed above). Two raters independently reviewed each article for inclusion and completed the quality appraisal tool for each. Discrepancies in ratings of the articles were discussed until consensus was reached or after discussion with a third rater. Raters did not review articles they had authored.

Figure 1. Search strategy and article appraisal.

Results

A total of 32 studies were identified for inclusion. Of those, 20 were considered multivariate and 12 descriptive; 24 used national level data and 8 provincial or territorial level data; 5 analyzed data from Quebec and 3 from Ontario; 1 used data from the First Nations component of the CIS–2008.

Table 2 shows the objectives, methods and quality assessment results of the included studies. Ninety-one percent of authors clearly described their research aims. In contrast, only 40% clearly defined all variables. Seventy-two percent of the studies provided clear inclusion criteria for cases and 72% described clear and appropriate sampling methods. Seventy-eight percent of analyses were considered appropriate to the authors’ research questions. The number of participants was considered sufficient 91% of the time. Analytical procedures and results were usually clearly explained (72%) and presented (75%). Eighty-one percent of conclusions were considered appropriate, and all but one study was judged to have been about a topic with clear applications for practice or policy.

TABLE 2. Characteristics of articles analyzing CIS and CIS oversample data.

Abbreviations:CIS, Canadian Incidence Study of Reported Child Abuse and Neglect; EIQ, Quebec Incidence Study of Reported Child Abuse and Neglect; FNCIS, First Naotins Component of the Canadian Incidence Study of Reported Child Abuse and Neglect; IPV, intimate partner violence; OIS, Ontario Incidence Study of Reported Child Abuse and Neglect.

The objectives of 28% of the articles were to do with particular types of maltreatment. Two were about neglect, three about emotional maltreatment, and two about exposure to IPV. One study was about exposure to IPV, hitting or neglect as sole concerns for investigation. None of the studies had objectives exclusively to do with physical abuse. One paper examined the frequency of joint police and child welfare investigations of sexual abuse cases compared with other maltreatment cases.

Table 3 summarizes the associations between independent, control and dependent variables measured in the articles presenting multivariate analyses. In this review, we define multivariate analyses as those that simultaneously measure multiple independent and/or dependent variables. For articles that described multiple models, only the final model is included for each dependent variable tested.

TABLE 3. Associations between independent, control, and dependent variables in the articles presenting multivariate analyses.

Abbreviations: FAE, fetal alcohol effects; FAS, fetal alcohol syndrome; IPV, intimate partner violence; R, reference group; -, no statistically significant relationship (p > .05); ↓, statistically significant negative relationship (p < .05); , statistically significant positive relationship (p < .05).

Note: Where more than one symbol appears in a cell, that article featured finer grained categories of the variable in question which have been collapsed forocncision and those finer categories had different relationships with the dependent variable.

^aThis variable combined physical and emotional harm.

^bThis variable combined physical and emotional harm.

^cThis variable was two or more moves.

Independent and control variables are grouped into five categories: maltreatment characteristics, child characteristics, caregiver characteristics, household characteristics and agency/referral characteristics. One article that used multivariate techniques was not included in the table because the analyses were classification and regression trees that could not be easily summarized in the structure of the table.

Most analyses that used maltreatment types (i.e. physical abuse, sexual abuse, neglect, emotional maltreatment and exposure to IPV) as independent variables included more than one maltreatment type. One study included only maltreatment variables representing different types of exposure to IPV, one included only physical abuse and two included only neglect. In addition, two studies considered emotional maltreatment as a dependent variable.

Researchers frequently used child age and sex as independent or control child-related variables. They tended to use some but not all the variables associated with child functioning concerns in their models, sometimes grouping all child functioning concerns together to create one dichotomous variable representing the presence of any of these concerns. The child functioning concerns span physical, mental, behavioural and developmental problems. Caregiver risk factors, especially substance abuse, mental health issues and lack of social support, were the most frequently used caregiver variables. Household hazards, overcrowding, household frequently runs out of money and frequency of moves were the most commonly used household variables. Finally, one or more case openings and a mother or parent as referral source were the most studied agency/referral variables.

The most commonly studied type of dependent variable, used in six articles, was the opening of a case for ongoing services. Researchers measured the associations of a variety of independent and control variables to do with maltreatment and child, caregiver, household and agency/referral characteristics with this outcome variable. The second most commonly studied outcome variable, used in four articles, was placement in out-of-home care. These articles included numerous variables to do with maltreatment, child, household and agency/referral characteristics. However, only two of four articles were about caregiver risk factors and those used few variables from the category. Other outcome variables that were studied, each in three or fewer articles, included retained cases for Aboriginal children, substantiated maltreatment, substantiated risk of maltreatment, emotional maltreatment, harm to the child, and child functioning concern variables including learning/developmental issues, physical disability/health conditions, emotional/behavioural problems, internalizing disorders, externalizing disorders, child functional impairment and police involvement in investigations.

Discussion

Our review summarized findings from the 32 identified articles on the quality and the relationships between variables in peer-reviewed journal articles using CIS data published since November 2011 and CIS oversample data. The majority of the reviewed articles had clearly stated research aims, used appropriate control variables (where studies were multivariate rather than descriptive), conducted appropriate analyses with sufficient sample sizes, had appropriate conclusions and were relevant to practice or policy.

The majority of the studies did not clearly define variables or provide rationales for the inclusion of the variables within analyses. Because most studies included a large number of variables, this may have been due to word limits imposed by publishers rather than a lack of planned rationale by the researchers. Often articles were judged to have unclear inclusion criteria and sampling methods when the authors had not explained how they selected cases for analysis from the full CIS sample and had provided only an overview of the CIS sampling methods. When the number of participants was considered insufficient, the researchers had usually used smaller subsamples of CIS cases for analysis. One strength of the CIS is that the sample size is large enough to conduct complex multivariate analyses, which is likely why this quality indicator was almost always rated positively. Analyses were frequently appropriate and results clearly presented; however, when they were not, it was usually because authors did not describe which analyses they were performing or did not clearly label tables. When analyses were gauged to be inappropriate to the research question, it was often because researchers used univariate analyses when multivariate approaches would have better addressed their research objectives. Studies that were multivariate in nature nearly always used appropriate control variables to account for potential confounding. Other studies used descriptive analyses to explore new topics, so that hypotheses could be generated and more sophisticated techniques could be used in future research. Conclusions were judged to be inappropriate when they extended beyond the scope of the analyses, such as when authors used causal language to describe an association. When an article was judged to be limited in relevance to practice or policy it was because the application of the findings was not clearly expressed, not necessarily because it had no potential application.

Regarding the associations between variables, some articles explored issues to do with substantiating maltreatment and service provision across the full sample of specific CIS waves or provincial subsamples, whereas other articles were concerned with analyses of subgroups of investigations classified by specific characteristics of children and families or specific characteristics of the investigated maltreatment or risk of maltreatment. Analyses tended to include child maltreatment types as variables when predicting opening for services, placement and child functioning concerns or harm but not when predicting substantiation. It is interesting that presence of harm was also not used to predict substantiation, although observable harm as evidence of maltreatment would presumably make maltreatment allegations easier for workers to substantiate.

This review has highlighted new areas relevant to policy. The multiple disciplines involved in child maltreatment were addressed in an article describing teachers’ reporting practices and the response to these by child welfare services.²³ This expands upon our knowledge of the reporting practices of different disciplines, in the same way that a similar article in the previous review described health care professionals’ reporting to child welfare agencies.³⁹ In addition, another article was about variables associated with joint police and child welfare worker investigations.³¹

Although changes to the CIS questionnaire are kept to a minimum between cycles, changes have been made to capture changes in practice. For example, CIS–2008 was changed to collect information on risk-only investigations.³ One of the articles focussed on untangling risk of future maltreatment from past events of maltreatment.¹⁸

Uses of the CIS data have evolved over time. In contrast to the previous review,⁷ in which 54% of studies were multivariate, 62.5% of the studies included in this review used multivariate approaches. Despite the increase in use of multivariate techniques, the objectives of most of the included studies did not include clear directional hypotheses as recommended.

The previous review⁷ found that physical abuse was the most frequently studied category of maltreatment as a main focus and exposure to IPV was the least. In the present review, nearly all the multivariate articles included either all forms of maltreatment or no forms of maltreatment as independent or control variables. The finding that more papers have focussed on exposure to IPV, emotional maltreatment and neglect is important for policy makers because neglect and exposure to IPV were the first and second most substantiated forms of maltreatment in the CIS–2008 and emotional maltreatment was the fourth.³ The three articles that focussed on neglect identified risk factors,^29,33 found a low presence of harm²⁹ and found that fathers are less likely to be present in cases of neglect.²⁸ The three articles that focussed on emotional maltreatment suggested that the increased specificity of definitions in the CIS–2008 helped differentiate between the occurrence and the risk of emotional maltreatment,³⁴ demonstrated that emotional maltreatment often co-occurs with other forms of maltreatment¹¹ and demonstrated that substantiated cases of single form emotional maltreatment are associated with more severe emotional impacts than other forms of maltreatment.¹⁰ The three articles that focussed on exposure to IPV described the characteristics of cases of exposure to IPV in single form, exposure to IPV and other maltreatment and exposure to other maltreatment;²⁵ suggested that exposure to different subtypes of IPV may have different associations with child functioning;⁶ and suggested that investigations of exposure to IPV or hitting only had lower- risk factors and were less likely to remain open compared with other investigations.³³

Another notable topic for recent uses of CIS data was investigations involving First Nations children. Five articles explored topics specifically related to First Nations children, including factors influencing overrepresentation at the investigation stage,³⁰ placement decisions^9,13,14 and differences from reports on non-Aboriginal children.³² Compared to investigations involving non-Aboriginal children, those involving First Nations children had a greater percentage of every caregiver/household risk factor (except health issues) identified by workers.³⁰ This could indicate a need for increased availability of family support services for First Nations families; however, the possibility exists that increased risk factors for these families are due to assessment bias.

Strengths and limitations

This review had a number of strengths. To increase the accuracy of judgments, all articles were reviewed by at least two raters who were not the authors of the article being evaluated. We used a standardized quality assessment tool used in an earlier review of CIS data usage⁷ to allow our findings to be compared to theirs. We included recent articles and articles based on oversample data, which broadened our scope.

This review also had a number of limitations. It did not include book chapters, theses, dissertations, government or agency reports, or unpublished manuscripts. It is possible that by excluding these sources we are failing to capture the breadth and depth of the research using CIS and CIS oversample data. We chose to exclude these sources, however, because they are not usually peer reviewed. Although oversample data exists for British Columbia, Alberta, Saskatchewan and the Northwest Territories, to our knowledge analyses stemming from these samples have not been published in peer-reviewed journals. Furthermore, we may have missed some articles from oversampling studies because data requests for these are not necessarily made through the Public Health Agency of Canada. In addition, because we only included published articles, our findings may suffer from publication bias (i.e. statistically significant variable associations may be overrepresented because nonsignificant findings are less likely to be published). Note, however, that most of the articles we included also presented findings of nonsignificant associations between variables. We include these nonsignificant findings in Table 3 to demonstrate the lack of relationships between some independent and dependent variables.

There are also limitations to our findings due to the nature of the CIS data. In reviewing these articles some of the limitations of the data were highlighted. Among these limitations are seasonal variations, the lack of independent verification of the data and the use of proxy-informants. In addition, CIS data only includes children who are reported to and investigated by child welfare agencies. Thus, selection bias may impact the population of children identified in the CIS as some children who experience abuse or neglect (for example, children from low-income families) may be more likely to be the subject of child welfare reports than others.³

Furthermore, the criterion validity of the variables within the CIS data may vary. For example, child welfare workers investigating reports of maltreatment or suspected maltreatment can be expected to be better trained and have greater expertise in assessing maltreatment characteristics compared to assessing child and caregiver functioning concern variables as identifying maltreatment would be their primary objective. As previously noted,⁴⁰ child functioning concerns may be underestimated because they are assessed using a checklist of issues known or suspected by the child welfare worker rather than a standardized systematic assessment, which would not be feasible in the study. In addition, not all child functioning items are relevant to different age groups. Analyses using the full age range of investigated children could have restricted ages of those included in the analyses to account for this; however, many did not.

Finally, all variables were measured at the time of the investigation. As such, it is impossible to know whether child maltreatment preceded child functioning concerns; nor can causality be established. These limitations are important for knowledge users and for researchers. The latter could study ways of further improving the quality of the data by considering the limiting parameters of the study.

Conclusions

This review has described the evolving nature of the application of the CIS and CIS oversample data to answer questions about substantiation, placement, provision of services and the impact of maltreatment on child functioning. It is clear that a multitude of factors determine these outcomes. Researchers using CIS data have recently focussed on categories of maltreatment (exposure to IPV, neglect and emotional maltreatment) that were previously inadequately studied, and examined factors specific to First Nations children.⁷ This review has highlighted newly investigated areas relevant to policy. It also suggests that data from the CIS oversamples has generally been underutilized in the peer-reviewed literature. In the future, researchers using CIS data may benefit from this analysis as it identified common pitfalls. The summary of research findings may help researchers identify unexplored topics in the CIS. Future research with CIS and CIS oversample data should continue to utilize sophisticated statistical modelling methods to take advantage of the breadth of information available to address research objectives.