Audit tool for assessing child sexual abuse prevention content in school policy and curriculum

Abstract

Although audit and feedback has not been well-used in social research, it is a useful method by which to answer research questions concerned with the notion of “best practice”, for example, whether a policy or program meets a required standard or benchmark. This paper draws on educational research conducted for a 5-year nation-wide public inquiry: the Australian Royal Commission into Institutional Child Sexual Abuse. The research comprised an audit of the strength and comprehensiveness of child sexual abuse prevention education policy and curriculum in 32 Australian school systems. The paper describes the development of the audit tool including its conceptual background, step-by-step process for identifying and synthesising evidence to generate the audit criteria and descriptors, and an explanation of how the tool was used. It also presents a succinct protocol for the research method, and critical commentary on the strengths and weaknesses of audit and feedback for social research.

1. Introduction

Violence including sexual violence during childhood affects educational outcomes and is associated with school absence, grade retention, dropout, and lower academic achievement (Fry et al., 2018; Romano et al., 2015). Child sexual abuse (before the age of 18 years) is experienced by approximately 10–20% of girls and 5–10% of boys worldwide on a spectrum from non-touch acts such as exposure and voyeurism through unwanted touching to rape (Pereda et al., 2009; Stoltenborgh et al., 2011). Perpetrators of child sexual abuse of girls and boys can be adults (such as household members, relatives, family friends, clergy, and teachers) and peers.

The most widely-used strategy for preventing child sexual abuse, to date, has been via school-based initiatives involving school policy and curriculum. School-based prevention is justified on several grounds: in most countries, primary (elementary) schools are a universal service providing access to learning for all children (Finkelhor, 2009; Wurtele, 2009); children spend substantial amounts of time at school, with unparalleled opportunity for learning (Pinheiro, 2006); schools are key agencies for addressing a range of issues influencing children's wellbeing (World Health Organization [WHO], 1998; 2015) including child sexual abuse; and school systems are becoming more attuned to the relationship between violence in childhood and educational outcomes (Fry et al., 2018; Romano et al., 2015). Effective school policies and curricula are, therefore, critical to ensuring that child sexual abuse prevention is appropriately prioritised, resourced, and implemented. However, very little is known about their scope and comprehensiveness.

1.1. Context and methods

Following growing awareness and concern about the abuse of children in institutions, the Australian Government conducted a five-year Royal Commission into Institutional Responses to Child Sexual Abuse (hereafter Royal Commission) (Commonwealth of Australia, 2017a) to investigate systemic failures to protect children from sexual abuse and recommend reforms to law, policy and practice. The Royal Commission instigated a program of independent research focused on answering eight research questions: (i) why does child sexual abuse occur in institutions? (ii) how can child sexual abuse in institutions be prevented? (iii) how can child sexual abuse be better identified? (iv) how should institutions respond where child sexual abuse has occurred? (v) how should government and statutory authorities respond? (vi) what are the treatment and support needs of victims and survivors and their families? (vii) what is the history of particular institutions of interest? and (viii) how do we ensure the Royal Commission has a positive impact? The research described in this paper fell within theme two.

We were appointed by the Royal Commission to conduct an evaluation of school-based policy and curriculum for child sexual abuse prevention applicable in Australian primary (elementary) schools — that is, schools serving children from pre-school to Year 6 or Year 7 depending on the jurisdiction (i.e., children aged approximately 5–12 years). Data were collected from all 8 Australian government and all 24 Catholic school systems, covering almost 90 per cent of Australian primary schools at the time (Walsh et al., 2017). This research was conducted alongside other complementary research projects with a wider purview, for example, reviewing efficacy of oversight bodies such as ombudsman offices, reportable conduct schemes, and children's commissions, and analyzing government oversight mechanisms such as schools' accreditation and health sector regulation (see for example, Mathews, 2017).

We had previously conducted a review of school policy and curriculum provision in 8 Government school systems in Australia (Walsh et al., 2013). The Royal Commission evaluation was to be conducted using “audit” methods (Jamtvedt, Young, Kristoffersen, O'Brien and Oxman, 2006). In designing the Royal Commission project we conceptualised “audit” as a type content analysis that could be used as both a research method and an intervention.

As a research method, audit can inform delivery of “best practice” by answering questions about whether an initiative meets a required standard or benchmark (Schwandt, 2011; Twycross and Shorten, 2014). Audit as a research method can be enacted in three ways. First, as an independent review of an intervention, program, service, or organisation against required standards or benchmarks (i.e., a third-party audit). Second, as a service-users’ review such as in client satisfaction surveys on key performance indicators (i.e., a second-party audit). Third, as a self-administered review in which an organisation assesses its own performance or progress against a set of quality criteria (i.e., a first-party audit) (Schwandt, 2011; Sen et al., 2013). As an intervention, audit can generate data and reports that act as levers or catalysts for organisational change. Empirical evidence shows that, when independent audit is accompanied by feedback, it can enhance the uptake of evidence into practice and improve professional practice (Colquhoun et al., 2017; Jamtvedt, Young, Kristoffersen, O'Brien and Oxman, 2006).

In a systematic review, Ivers et al. (2012) showed that audit and feedback approaches have been used primarily in the field of healthcare. For example, by assessing professional practice (such as clinician referral of patients for mammography or implementation of blood pressure monitoring protocols), comparing results to best-practice guidelines, and providing feedback to clinicians or their on their performance. Audit and feedback is proposed to “work” via changing clinicians’ awareness and beliefs about the clinical practice and its benefits, adjusting social norms in clinical work groups, building clinician self-efficacy and focusing clinician attention on the specific micro-tasks that comprise the overall clinical practice (Ivers et al., 2012; Colquhoun et al., 2013).

1.2. Purpose

The purpose of this paper is twofold. We describe the development and use of an audit tool as a research instrument. We begin by scoping definitions, before detailing the ten-step audit tool development process, and explaining how the tool was used to assess school system performance in child sexual abuse prevention. We also explain the audit and feedback method and critically discuss the strengths and weaknesses of the audit approach for social research. We do not report on audit findings as these are published in detail elsewhere (Walsh et al., 2013; Walsh et al., 2017). The audit tool development process, however, has not previously been reported in sufficient detail to enable replication.

1.3. Definitions

For the purpose of this research, Policy was defined as a public statement expressing what an educational authority intends to do — or to not do — to address a problem or a set of problems (Birkland, 2005; Pal, 2009), in this case, child sexual abuse. The term policy refers to the substance of official written texts — policy documents. Curriculum was defined as the core knowledge, understandings, skills, capabilities, and dispositions students should learn and acquire as they progress through school (Australian Curriculum, Assessment and Reporting Authority [ACARA], 2013). The Australian Curriculum is developed and published by an independent statutory Commonwealth government authority. Each school system or education authority is responsible for implementing the Australian Curriculum, including decisions about its sequencing, timing, teaching practices, materials, resources, and assessments (ACARA, 2016).

The unit of measurement in the study was the school system or overarching administrative entity, such as a state or territory government department of education, or a Diocesan Catholic education office. At present, state and territory government (or public) school systems in Australia serve approximately 65.4% of the school-aged population (range: 59.6%–71.8% in the individual states and territories). Catholic school systems serve approximately 20.2%, with independent schools serving the remainder (14.4%) (Australian Bureau of Statistics [ABS], 2017). There are eight government school systems, one in each of Australia's six states and two territories, 24 Diocesan Catholic school systems, and numerous independent school authorities, the largest of which are the Lutheran, Anglican, and Seventh Day Adventist school systems (Independent Schools Council of Australia, 2016). The latter group (independent schools) were excluded from the audit.

1.4. Audit tool development process

The School Policy and Curriculum Audit Tool was developed in ten steps, in two iterations spanning several years as shown in Fig. 1. Steps 1–3 were completed in an earlier review of school policy and curriculum across Australian state and territory Departments of Education (Walsh et al., 2013). Steps 4–10 were completed as part of the Royal Commission research. As a methodological contribution, stepwise documentation of the audit tool development provides technical guidance for researchers on how to generate and use a consolidated set of evidence-based evaluation criteria for school policy and curriculum. Over time, we developed a design process that was sufficiently rigorous, transparent, replicable, and acceptable to audit stakeholders including the research funders (i.e., the Royal Commission), academic and industry peer reviewers, and school authorities.

Fig. 1.

10-step process for audit tool development.

The process, however, was not without its challenges. In the absence of established guidelines for audit tool development, we began by drawing on frameworks for the development of consolidated guidelines to improve healthcare (Eccles et al., 2012; Shekelle et al., 2012; Woolf et al., 2012). Handbooks for guideline development stress the importance of identifying the most appropriate sources of evidence from which evaluation criteria can be extracted. Consistent with our research brief, guideline development methods also incorporate attention to the needs of vulnerable groups and rigorous stakeholder review (World Health Organization, 2014).

1.5. Step 1

Step 1 involved conducting a rapid evidence review, a variant of the traditional systematic review completed in a shorter time (Ganann et al., 2010). We searched for existing guidelines, standards, best practice frameworks, and recommendations for school system-based child sexual abuse prevention policy and curriculum. Nine EBSCO Host electronic databases were searched (CINAHL; Criminal Justice Abstracts; E-Journals; ERIC; Medline; Psych Articles; Psych Info; Social Work Abstracts; and Violence and Abuse Abstracts) using the search strategy shown in Fig. 2.

Fig. 2.

Search terms and strategy.

Searches yielded 2121 records after duplicates were removed. Titles and abstracts were screened by two research assistants working independently using inclusion criteria: (i) date (published from January 1990 onwards); (ii) language (published in English); (iii) type of paper (guideline, framework, position paper, research report, research review or synthesis, statement of recommendations); (iv) topic (concerned with “best practice” in policy or curriculum); and (v) focus (primary prevention of child sexual abuse or primary prevention in fields relating to child sexual abuse prevention e.g., sexual assault, sexuality education). This resulted in 15 potentially-relevant records, however, after full-text review none met the inclusion criteria.

We then hand-searched reference lists of the 15 papers and conducted keyword searches of relevant organisations for grey literature (e.g., research reports not indexed in academic databases). Search terms were congruent with the search strategy above. These searches yielded three primary prevention frameworks meeting inclusion criteria. These were: the National Framework for Health Promoting Schools (Australian Health Promoting Schools Association [AHPSA], 2000); the National Safe Schools Framework (Ministerial Council for Education, Early Childhood Development and Youth Affairs [MCEECDYA], 2011); and the Spectrum of Prevention (Cohen and Swift, 1999).

1.6. Step 2

From the three frameworks, we extracted 10 evaluation criteria with a total of 17 descriptors (guiding questions). These were transferred to an evaluation matrix (i.e., a data collection template) against which we assessed each Australian state and territory education system in a “policy and curriculum scan” (Walsh et al., 2013, pp.649–650). The 10 criteria (with example guiding questions) were: (i) base child protection policy (e.g., do child protection policies exist?); (ii) specific child sexual abuse prevention education policy (e.g., does a specific child sexual abuse prevention education policy exist?); (iii) commitment (e.g., is teaching of child sexual abuse prevention education optional or compulsory?); (iv) location in the curriculum (e.g., where is child sexual abuse prevention education located in the primary school curriculum?); (v) curriculum support (e.g., what curriculum support is offered to schools and/or teachers in the teaching of child sexual abuse prevention?); (vi) pedagogical support (training) (e.g., what training is offered to schools and/or teachers in the teaching of child sexual abuse prevention?); (vii) assessment (e.g., is student learning in child sexual abuse prevention education assessed?) (viii) parental permission (e.g., is parental permission required for children's participation in child sexual abuse prevention education at school?); (ix) information for parents and communities (e.g., what information is provided to parents and/or communities about child sexual abuse prevention education at school); and (x) partnerships (e.g., what partnerships are evident with relevant community services or agencies when schools teach child sexual abuse prevention education?) (see Walsh et al., 2013, pp.653–655).

1.7. Step 3

In 2011, twice, at 3-monthly intervals, we systematically searched Australian state and territory education department websites using a standardized set of guiding questions to determine the extent to which each of the 10 criteria were met. This study was part of an Australian Research Council Discovery Project (ARC DP1093717) that uncovered “substantial inequity in Australian children's access to learning opportunities in child sexual abuse prevention education” in primary (elementary) school (Walsh et al., 2013, p.649).

1.8. Step 4

The audit method requested by the Royal Commission required adaptation of the original evaluation matrix to address their Terms of Reference (Commonwealth of Australia, 2012). For example, to ensure a focus on policy and curriculum provision for children at greater risk of child sexual abuse such as those from culturally and linguistically diverse backgrounds, children with disability, and Indigenous children. Additional evidence sources were needed, yet our earlier searches had yielded little empirical research on “best practice” for school-based policy and curriculum in the area of child sexual abuse prevention. Using a broader view of evidence-informed prevention, we drew on Puddy and Wilkins (2011, p.4) “framework for thinking about evidence” as three distinct but overlapping spheres comprising: the best available research evidence; experiential evidence represented by practice-based knowledge and experience; and contextual evidence. We then searched institutional websites, including those of peak bodies (e.g., WHO, UNICEF, UNESCO, OECD); non-government organisations (e.g., Plan International, Save the Children, World Vision) and government departments of education, health, and welfare (for Australia, Canada, the European Union, New Zealand, the United Kingdom, and the United States) using search terms consistent with those used previously. We also circulated requests for relevant evidence to email listservs and networks. These strategies yielded five further sources: four guidelines or standards, and one research review.

Table 1 shows all evidence sources used in the revised audit tool, including author, country of origin, publication title, and applicable sections. Each of the sources contained a set of recommended “best practice” approaches labelled variously as strategy levels, key action areas, key elements, criteria, standards, guidelines, characteristics, or principles. Importantly, there was relative consensus across these sources in the areas relating to school policy and curriculum. A detailed document mapping the evidence sources against each of the revised criteria can be obtained from the authors.

Table 1.

Frameworks, guidelines and standards, research reviews and syntheses used in developing the audit tool.

	First author (year)	Country	Title	Sections most applicable
Frameworks for primary prevention	Australian Health Promoting Schools Association (2000)	Australia	A national framework for health promoting schools (2000–2003)	Key action areas 1, 2, 3, 5 & 6
	Cohen and Swift (1999)	United States of America	The spectrum of prevention: Developing a comprehensive approach to injury prevention	Spectrum levels 1, 2, 3, 4 & 6
	Ministerial Council on Education, Early Childhood Development and Youth Affairs (2011)	Australia	National safe schools framework: All Australian schools are safe, supportive and respectful teaching and learning communities that promote student wellbeing	Key elements 1, 2, 3, 4, 6 & 9
Guidelines & standards	Flood et al. (2009)	Australia	Respectful relationships education: Violence prevention and respectful relationships education in Victorian secondary schools	Criteria 1, 3 & 4
	Carmody et al. (2009)	Australia	Framing best practice: National standards for the primary prevention of sexual assault through education	Standards 3, 4 & 6
	National Center for Missing and Exploited Children (1999)	United States of America	Guidelines for programs to reduce child victimization: A resource for communities when choosing a program to teach personal safety to children.	Guideline (presentation) 2, 3, 4 & 5 Guideline (awareness) 4
	United Nations Educational, Scientific and Cultural Organization (2009)	France	International guidelines on sexuality education: An evidence informed approach to effective sex, relationships and HIV/STI education	Characteristics 3, 4, 5, 9, 10, 11, 12 & 15
Research reviews& syntheses	Nation et al. (2003)	United States of America	What works in prevention: Principles of effective prevention programs	Principles 1, 2, 5, 7 & 9

1.9. Step 5

The Royal Commission required that the research tool was peer-reviewed by stakeholders and experts prior to data collection. The revised audit tool and proposed data collection methods were peer-reviewed by several of the Royal Commission's policy and research leaders, two experienced researchers with expertise in the field of school-based child sexual abuse prevention. Feedback suggested extra guiding questions for some criteria, and refining others. Suggestions were incorporated, and the audit tool was finalised.

1.10. Step 6

The final School Policy and Curriculum Audit Tool comprised 10 criteria, each with a set of descriptors (guiding questions) for data collection. Guiding questions ranged from eight questions for criteria 1 to two questions for criteria 4, with a total of 39 questions across the ten audit criteria as shown in Table 2. The audit tool was used as a standardised matrix for data collection.

Table 2.

Audit tool used for the Royal Commission's audit of school policy and curriculum: Criteria and guiding questions.

Criteria	Guiding questions
1. Base child/student protection policy	•Does a child protection policy exist? Identify the policy by name and provide the URL, document identification details, date of publication, date policy was last updated and total number of pages in the policy. •Is there a navigable pathway to the policy from the school system home page? Record the pathway from the home page to the policy, and the number of steps involved. •Does the home page search feature locate the policy using the terms 'child protection', 'student protection' or 'child abuse' within the first five to 10 hits? •Can the policy be downloaded as a single document? •Does the policy need to be read in conjunction with other documents? If so, identify these documents by title and record the number of pages in each document. •What specific guidance is provided for situations of sexual abuse by individuals employed by or volunteering at the institution? Identify section numbers and page numbers to pinpoint where this information is provided. Record broad categories of information provided. •Is the teaching of child sexual abuse prevention education specified in the policy? Insert representative quotes and source (including section number and page number). •What related documents are listed? Record the total number of documents listed and record the titles of the documents most relevant to child protection.
2. Specific policy on child sexual abuse prevention education	•Is there a specific policy on child sexual abuse prevention education? Identify the policy by name and provide the URL, document identification details, date of publication, date the policy was last updated and total number of pages in the policy. •Does the policy specify evidence-based approaches? List any research cited in the policy. Does the policy specify updating in response to new research? •Does the policy address the potential for child sexual abuse to occur in child-related institutions (such as schools) by individuals employed by or otherwise engaged by the institutions, and by other children within the institutions? If so, describe. •Is child sexual abuse prevention education optional or compulsory? Describe the approach. •If compulsory, how do schools report on compliance? Describe the reporting mechanisms.
3. Location in the curriculum	•Where is child sexual abuse prevention education located in the primary school curriculum (that is, which parts of the school curriculum relate to child sexual abuse prevention education)? Indicate if this is explicit or implicit. Nominate learning areas/subjects, strands and learning outcomes. Refer to (i) school curriculum, (ii) specific child sexual abuse prevention education curricula, (iii) other child sexual abuse prevention program curricula (internal), and (iv) other child sexual abuse prevention education curricula (external). •Is child sexual abuse prevention education named in the primary school curriculum? •What recommendations are made for specific programs or for the timing, frequency or intensity of sexual abuse prevention education? •What recommendations are made for personnel in the teaching of child sexual abuse prevention education (for example, who will teach it and why; what expertise or qualifications are required)?
4. Inclusion, relevance, and cultural sensitivity	•What strategies are recommended for schools and teachers in differentiating the teaching of child sexual abuse prevention education for different groups of children (that is, for children of different ages and genders, children of culturally and linguistically diverse backgrounds including refugee and asylum-seeking backgrounds, Australian Indigenous children, children with disabilities, children in out-of-home care, and same-sex attracted and gender-questioning children)? •Are curricula and/or programs presented in languages other than English? If so, indicate which language(s).
5. Curriculum support	•What curriculum support is provided to schools and/or teachers in the teaching of child sexual abuse prevention education? •Which specific programs or materials are suggested for use? •What agencies are involved in providing child sexual abuse prevention curriculum support to teachers? Are contractual or other arrangements in place? •Do materials and resources acknowledge the potential for institutional child sexual abuse to occur and provide support for teaching about this?
6. Pedagogical support	•Is training specified in the child protection policy? •What training is provided to school staff in the teaching of child sexual abuse prevention education? Describe training provisions, formats, duration and frequency. •Which specific internal/external training agencies are involved and how? •Which specific training programs or materials are used and why? •Are contractual or other arrangements in place between school systems and other agencies (for example, preferred providers) for the delivery of staff training? If so, describe.
7. Assessment & evaluation	•Is student learning about child sexual abuse prevention assessed? If so, how? •Has the child sexual abuse prevention education/curriculum been evaluated? If so, who conducted the evaluation? When? How (what approach was used)? What data were collected and from whom? •Were evaluation findings implemented?
8. Information for parents and communities	•Is there a parent navigation tab visible on the website home page? •Is information provided to parents about institutional responses to child sexual abuse, including processes for reporting known or suspected institutional child sexual abuse? •Is information publicly available to parents/communities about child sexual abuse prevention education delivered at school? •What information is directed towards parents specifically about their involvement in child sexual abuse prevention education? •Is parental permission required for children's participation in child sexual abuse prevention education at school?
9. Information for children and young people	•Is there a student navigation tab visible on the website home page? •What information is provided to children and young people about institutional responses to child sexual abuse, including processes for reporting known or suspected institutional child sexual abuse? •What information is provided to children and young people about child sexual abuse prevention education delivered at school?
10. Partnerships with community services and agencies	•What reference is made in policy and procedures documents (for child protection, child sexual abuse prevention and/or mandatory reporting) to partnerships with support-based external agencies or community services for referrals arising from the teaching of child sexual abuse prevention in schools? •What reference is made on departmental or diocesan websites to partnerships with support-based external agencies or community services for referrals arising from the teaching of child sexual abuse prevention in schools? •What reference is made by departments or dioceses, in submissions to the Royal Commission, to partnerships with support-based external agencies or community services for referrals arising from the teaching of child sexual abuse prevention in schools? •What agencies are nominated to receive referrals in relation to child sexual abuse? •Are the agencies predominantly government or non-government agencies? •Are the agencies nominated of direct relevance to child sexual abuse, prevention education, disclosures, or counselling support?

1.11. Step 7

From January to April 2014, data were collected from 32 Australian government and Catholic school systems serving primary (elementary) school students (Walsh et al., 2017). A total population sampling frame was used (Daniel, 2012; Lund Research, 2012) in which we purposively sampled all Australian state and territory government departments of education (eight departments) and all Australian Catholic education offices (24 offices representing 28 Catholic dioceses). The audit covered almost 90 per cent of Australian primary schools at the time (ABS, 2014).

As independent third-party auditors, we used the internet as the primary research medium for data collection (Lee et al., 2008). Data were collected by manually searching the publicly accessible areas of each of the 32 school system websites, working through each of the 10 criteria, one descriptor at a time. To capture as much relevant information as possible and to minimise the potential for bias, websites were searched at least twice, once by the first author and again by either the second or third authors, working independently. Data were abstracted into data extraction templates in MS Word and Excel. Entries for each descriptor were compared, and discrepancies resolved via discussion and search repetition where necessary. For completeness and transparency, website URLs and other details were recorded verbatim into the data extraction templates (see Walsh et al., 2017, Volume 3, pp.231–839).

We used qualitative content analysis (Hsieh and Shannon, 2005; Krippendorff and Bock, 2009) to classify large volumes of text into a well-organised set of meaningful categories for each of the guiding questions. Data summary tables were developed to neatly display data for guiding questions in each criterion (for examples see Walsh et al., 2017, Volume 2, pp. 73–230). Data from all 32 school systems were pooled and quantitative counts (e.g., frequencies and proportions) and qualitative descriptions (e.g., summary statements) were provided where appropriate. Exemplars were used to illustrate “good practice”, and incidental findings were reported (for examples see Walsh et al., 2017, p 235, 244, 247). A draft report was prepared and submitted in late 2014.

1.12. Step 8

In 2015, in line with procedures for using audit as an intervention, the Royal Commission provided state and territory Departments of Education and Diocesan Catholic education offices with a copy of the draft research report and sought their written responses, including submission of any materials that they believed had not been, but should be, included in the audit (these were known as supplementary materials). Responses were received from 28 of the 32 school systems. Some responses offered general comments. Others provided specific comments against some or all of the audit criteria. Yet others provided evidence of policy and curriculum improvements undertaken since data were collected.

1.13. Step 9

Supplementary materials were assessed by the first author and categorised as “within scope” (e.g., materials had been included in the original audit), “out of scope” (e.g., materials had been developed since the audit period), or “unclear” (e.g., materials were not publicly accessible to the researchers). Of note, many of the school systems had conducted their own “self-audit” against the audit criteria to verify the research results, thereby capturing comparative data in their responses. From a methodological standpoint, Step 9 enabled the final report to provide a fair and reasonable assessment of all measures taken by Australian state and territory departments of education and Diocesan Catholic education offices to address child sexual abuse prevention via school policies and curricula during the audit period.

1.14. Step 10

Following the review of supplementary material, the draft report was revised and updated. The full audit report, complete with raw data and summary tables, was published on the Royal Commission's website in December 2017 (Walsh et al., 2017) and forms the basis for several recommendations in the Final Report relating to schools and child safe organisations (Commonwealth of Australia, 2017a, 2017b, 2017c).

2. Discussion

In this paper we have described the process undertaken to develop a School Policy and Curriculum Audit Tool that can be used to assess the strength and comprehensiveness of child sexual abuse prevention in school system policies and curricula, and to prompt change via the provision of feedback. Our audit focused on school policy and curriculum and was heavily weighted towards curriculum. Now that the Royal Commission has concluded its inquiry, it is patently clear that school-system policies must be substantially strengthened to incorporate a much broader range of strategies for creating child safe organisations (Commonwealth of Australia, 2017b, pp.22–27) beginning with embedding accountability mechanisms into organizational leadership, governance and culture via stronger regulation, monitoring and oversight (Commonwealth of Australia, 2017a; Mathews, 2017). For example, codes of conduct, providing guidelines for school staff on expected behavioral standards and responsibilities towards children, were identified by the Royal Commission as vital for child safety. If we were developing the audit tool now, we would expand Criterion 1 to incorporate these strategies. Congruent with the Royal Commission's findings, in addition to victim prevention strategies, we would also add indicators for offender prevention. Our research advances evaluation in the field of school-based prevention by offering school systems a methodology for developing a flexible tool that can be used by independent evaluators and/or school systems themselves to assess and monitor progress towards child safety goals.

Although evaluation is a hallmark of democratic societies, and arguably also of institutions that aspire to learn from their mistakes and improve (Dahler-Larsen, 2016), audit by a third-party presents many challenges. First among these is the “adversarial potential” of audit processes (Power, 1999, p.126), an artefact of its methods. Auditors necessarily assess what is relative to what should be. Thus they typically conduct research independently of the auditee. Auditors place strict protocols around documenting of evidence and make use of research tools in ways that can clearly identify the gap between optimal and actual provision of services (Hut-Mossel et al., 2017; Schwandt, 2011). The potential for auditors to identify necessary improvements can contribute to an uncomfortable or even adversarial relationship between auditor and auditee, especially when institutions have already invested in improvements prior to or during the audit period.

The benefits of audit processes cannot be presupposed. One criticism of the “audit society” (Power, 1999) is that audits promise accountability and transparency to stakeholders when dialogue may well achieve desired outcomes (Power, 1999, p.127). This point was not lost on us as audit researchers. To combat the threat of “big stick diplomacy” we drew on the concept of school systems as “learning organisations” in relation to child sexual abuse prevention. Learning organisations are those in which everyone can be encouraged to learn, and that learning is valued, promoted, and reinforced by organisational leaders (Garvin et al., 2008, p.1). Although we were surprised and at times frustrated by inadequacies we encountered in our searches which leave children vulnerable and teachers lacking information, we were also hopeful. Some of the websites we audited for information evolved during the audit period and we witnessed information becoming easier to find: policies and processes were updated, and curricula were revised and implemented. That schools live and breathe in constant cycles of quality assurance and improvement may in actual fact auger well for child sexual abuse prevention.

In the study conducted for the Royal Commission, a feedback loop was added to serve not only as a catalyst but as a mechanism for school system improvement. In this way, the audit was not solely “one-way surveillance” (Power, 1999, p.134). By levering a legitimate requirement for the audited school systems to publicly account for their actions, the audit held the potential to inform change at a systems level. Such accountability ensures school systems are responsive to their constituents. In the light of the serious abuse and the individual and collective atrocities uncovered by the Royal Commission, Australian parents and children will rightly require that organisations account for their progress against the benchmarks in our report and the Royal Commission's recommendations. Although the school policy and curriculum audit may have applied some threat of external sanction or pressure to improve, the real incentive to improve practice in future will be a moral and ethical one. Our hope is that this audit tool “exists to become obsolete” (Schwandt, 2009, p.202). That is, the tool does its work via providing a catalyst for change and that change is then be self-sustained, for example, via ongoing self-administered first-party review (Schwandt, 2011; Sen et al., 2013). Increasing self-audit capability and school systems that embrace their role as “learning organisations” (Garvin et al., 2008, p.1) will decrease the need for third-party review. Indeed, the National Statement of Principles for Child Safe Organisations, which emerged from the Royal Commission's policy and research work (Valentine et al. 2016) states:

“child safe organisations seek to regularly improve their delivery of services and their operations. They also conduct reviews to ensure that organisational policies and procedures are being implemented by staff and volunteers. The participation and involvement of staff, volunteers, children and young people, families and community mentors in these reviews will strengthen the organisation's child safeguarding capacities. This includes the importance of reporting on the finding of reviews, and sharing good practice and learnings on a regular basis. Regular reviews ensure that organisations address new challenges or concerns that arise” (Australian Human Rights Commission, 2017, p.15).

Additionally, research shows that audit and feedback will be more likely to be effective when feedback is provided by senior organizational leaders or respected influencers, delivered in written and verbal modes more than one time, explicitly identifying behaviours that should be stopped or decreased, and providing specific alternative goals and action plans (Colquhoun et al., 2017; Ivers et al., 2012).

2.1. Limitations and strengths

There are limitations in the audit tool development process that must be acknowledged. First, in identifying sources of evidence (in Steps 1 & 4), it was difficult to adhere strictly to a systematic approach as many of the sources we identified for use in the audit tool, were not indexed in academic databases. It is important that researchers and organisations developing frameworks, guidelines, and other best practice tools, ensure their reports can be found using standard search terms in frequently-used academic databases. Emerging protocols for guideline development may help to address this problem (for example, WHO, 2014). Second, we did not conduct quality appraisals for the eight sources of evidence that were identified via hand searching in the rapid review (Steps 1 & 4), as the tools for quality appraisal of empirical studies (e.g., the Cochrane Collaboration's tool for assessing risk of bias in intervention studies) were not suitable. We have since become aware of the Appraisal of Guidelines for Research and Evaluation (AGREE) II tool (Brouwers et al., 2010) which, although developed for healthcare settings, has several criteria that could retrospectively be applied to the eight sources of evidence to strengthen the audit tool's empirical basis. These criteria include: “the criteria for selecting evidence are clearly described”, “there is an explicit link between the recommendations and the supporting evidence”, and “the guideline [or framework] has been externally reviewed by experts prior to its publication” (p.841). Third, although at least two researchers collected data from school system websites, we did not calculate inter-rater reliability as it was not feasible with such a large volume of data. To strengthen the audit tool's empirical basis, in future research inter-rater reliability could reasonably be collected on a random sample of guiding questions for each criteria.

There are also several limitations of the third-party audit approach in relation to data collection that must be made clear. First, school system websites were highly variable in appearance, accessibility, organisation, and detail. It was clear that varying levels of investment had been made in this technology as a child safety resource, with state and territory government departments of education having the most comprehensive websites. Second, although extensive searches of school system websites were conducted, we found that some potentially useful links directed visitors to password-protected school intranets, from which we were unable to collect data. It is our view that certain essential categories of information (such as school system policies, and information for parents about child sexual abuse prevention) should be made publicly available outside of a restricted intranet environment, not only to benefit future audits, but to fully inform school communities about school systems' intended actions for preventing and responding to child sexual abuse. Third, the Australian state government departments of education and Diocesan Catholic education offices provided varying levels of detail in response to the Royal Commission's requests for response to the draft audit report and provision of supplementary material (Steps 8 and 9). Several school systems commented on difficulties in allocating resources to complete the audit response. Fourth, the Australian state government departments of education and Diocesan Catholic education offices did not have the opportunity to shape the audit criteria at the project outset, and the audit may have been conducted differently had they been able to provide such input.

Finally, the strength of the approach used in the audit lies in its broad investigative scope and depth, and its reporting comprehensiveness and transparency. As Wright et al. (2017) write, the statutory powers of the Royal Commission enabled deep and compelling research to be undertaken, shedding light on enablers of institutional child sexual abuse in ways that would otherwise have been impossible. Schools are one of the institutions that have failed to prevent child sexual abuse, even in the recent past. The unparalleled access to information and level of detailed analysis that was possible in this present research under the Royal Commission's jurisdiction would simply not have been possible via other means. Although some of the audit criteria were amenable to direct observation without the assistance of school systems (i.e., via publicly available material available on school system websites), obtaining other information was facilitated by the Royal Commission, including information on the internal evaluations of school curricula, which required school systems' additional cooperation.

3. Conclusions

School system policy and curriculum can serve as a measure of education authorities' symbolic and material actions in relation to child sexual abuse prevention. The School Policy and Curriculum Audit Tool (Walsh et al., 2017) provides a valid and practicable means of auditing school systems for the strength and comprehensiveness of policy and curriculum for prevention of child sexual abuse. Future research should test the tool in different school systems to determine how using such tools influences school-system improvement related to child sexual abuse prevention. Audit and feedback need not be mandated to effect change, though their positive effects may be greater when institutions’ staff are actively involved in the audit process, with defined roles and responsibilities for realising change (Jamtvedt et al., 2006). In any case, this research demonstrates the potential of an audit tool to collect meaningful data on child sexual abuse prevention policy and curricula, rendering the audit tool itself and the results of great benefit to the institutional governance of schools in Australia. Using this audit tool we have assessed the approaches taken by government and Catholic school systems to address child sexual abuse prevention, thereby identifying where efforts to improve or change practice are most needed.

Declarations

Author contribution statement

K. Walsh: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Wrote the paper.

L. Brandon, L. A. James Kruck: Performed the experiments; Analyzed and interpreted the data; Wrote the paper.

Funding statement

This work was supported by the Australian Attorney General's Department via the Royal Commission into Institutional Responses to Child Sexual Abuse. Related research was funded by the Australian Research Council's Discovery Projects Scheme under grant number DP1093717.

Competing interest statement

The authors declare no conflict of interest.

Additional information

Supplementary content related to this article has been published online at https://www.childabuseroyalcommission.gov.au/sites/default/files/file-list/research_report_-_audit_of_primary_school-based_sexual_abuse_prevention_policy_and_curriculum_-_prevention.pdf.