A Novel Scale to Communicate Perceived Likelihood of Child Sexual Abuse

Abstract

Background and Objective:

Child abuse pediatricians (CAPs) are often asked to determine the likelihood that a particular child has been sexually abused. These determinations affect medical and legal interventions, and are important for multi-site research. No widely accepted scale is available to communicate perceived sexual abuse likelihood. In this study, we measure intra- and inter-rater reliability of a 5-point scale to communicate child sexual abuse likelihood.

Methods:

We developed a 5-point scale of perceived likelihood of child sexual abuse with example cases and medical-legal language for each risk category. We then surveyed CAPs who regularly perform sexual abuse evaluations using the abstracted facts of 15 actual cases with concern for sexual abuse. A subset of participants rated the same vignettes again, one month later.

Results:

Of 512 invited participants, 240 (46.7%) responded, with 145 (28.3%) indicating that they regularly perform sexual abuse evaluations, 116 initially completing all 15 vignettes, and 36 completing repeat ratings at least one month later. The scale showed consistent stepwise increase in mean perceived likelihood of abuse and intention to report for each increase in scale rating. Inter-rater agreement was substantial (Fleiss’ weighted kappa 0.64) and test-retest reliability among 36 participants was almost perfect (Cohen’s kappa = 0.81).

Conclusions:

We introduce a scale of perceived sexual abuse likelihood that appears to reflect CAPs’ perceptions and intention to report. This scale may be a reasonable metric for use in multi-center studies. CAPs demonstrated substantial inter- and intra-rater reliability when evaluating sexual abuse likelihood in case vignettes. While this scale may improve communication of sexual abuse likelihood among experts, its examples should not be used as a legal standard or a clinical criterion for sexual abuse diagnosis.

Introduction

In cases with concern for child sexual abuse, diagnostic uncertainty may remain even after all evaluation is completed.¹ In such a case it would be inappropriate to report that sexual abuse is either “confirmed” or “excluded” because the practitioner’s actual impression lies somewhere between these extremes of confidence. In such cases, accurate communication of the perceived likelihood of abuse is important to select appropriate medical, social, and legal interventions. Perceived abuse likelihood is also an important metric for research studies that compare diagnostic decisions between providers and centers.^2–4

Some prior research in child sexual abuse has used endpoints such as criminal conviction,⁵ pregnancy,⁶ confession,⁷ and video-recorded abuse⁸ that are highly specific to sexual abuse. Each of these endpoints, however, is rare. The patients experiencing them may not be representative of all children experiencing sexual abuse.

Researchers studying physical abuse often use opinions of an examiner or a multidisciplinary team as an outcome.^9–14 Many of these studies use a seven point scale⁴ for physical abuse likelihood. Studies investigating the recognition and interpretation of physical findings in sexual abuse have also used various 3 to 7 point ordinal scales^5,7,15–19 to express examiners’ opinions regarding sexual abuse likelihood. However, because the majority of sexually abused children have no physical findings, these scales do not express an examiner’s overall opinion regarding sexual abuse likelihood.

Adams and colleagues have developed widely-accepted guidelines that group physical or laboratory findings into multiple levels of specificity for sexual abuse.²⁰ These guidelines, however, focus on the interpretation of individual physical and laboratory findings as opposed to a global assessment of sexual abuse likelihood. Earlier work by Adams and colleagues proposed, but did not validate, a scale for overall likelihood of abuse that is absent from that author’s later guidelines.^5,21,22

An ideal scale of sexual abuse likelihood would communicate multiple levels of sexual abuse likelihood, rather than a binary result. If a scale is to be useful for communicating between providers, it must be interpreted consistently. That is to say, providers who determine, or who are told, that a child has “high likelihood of abuse” must agree on what range of likelihood counts as “high.” An ideal scale would be capable of assessing the full range of likelihood expressed by child abuse pediatricians (CAPs), would be easy to use, and would be correlated with interventions recommended by those providers. Therefore, our objective is to test if our scale of sexual abuse likelihood met these criteria.

Methods

We surveyed CAPs who regularly evaluated children for sexual abuse using vignettes derived from real cases of children who had been evaluated for sexual abuse. The Institutional Review Board at the Medical University of South Carolina determined that this project was exempt from IRB supervision.

Scale development

We created the 5-point scale shown in Table 1. For each level on the scale, several examples of clinical scenarios and medical-legal summary language are given to guide scale users. The clinical scenarios are intended as illustrations rather than strict criteria to replace clinician judgment for the diagnosis of sexual abuse. The scale’s design is informed by a review of child sexual abuse evaluation forms from 19 child abuse centers across the country and was refined by expert feedback. To ensure that the selected case vignettes did not artificially influence scale design, the scale was completed prior to selection of the case vignettes.

Table 1.

A scale for communicating the likelihood of sexual abuse

Rank	Name	Explanation	Examples
1	No Sexual Abuse Concerns	While no evaluation can completely exclude abuse, the evaluation has not identified factors that raise concern for abuse, or has identified non-abusive explanations for concerning signs or symptoms.	• No disclosure of abuse and • Isolated, non-abusive sexual activity (e.g., peer-peer sexual play) • Physical symptoms explained by mimic (e.g. urethral prolapse, strep)
2	Low Likelihood of Sexual Abuse	The identified behaviors, signs, and symptoms are non-specific for abuse, and are frequently seen in children who are not abused.	• No disclosure of abuse and • Isolated genital redness, masturbation, or bedwetting • Exposure to registered sexual offender
3	Indeterminate	The identified signs and symptoms are concerning for sexual abuse, but could be explained by non-abusive causes.	• Inconsistent Disclosure • Disclosure is a single sentence or phrase, but child does not provide additional details. • Disclosure to parent not repeated in forensic interview • Problematic sexual behaviors in a preadolescent without a disclosure of sexual abuse. • Anogenital condyloma in a child older than 3
4	High Likelihood of Sexual Abuse	The identified signs or symptoms are strongly concerning for sexual abuse, and are unlikely to result from non-abusive causes.	• Clear, narrative history of sexual abuse and surrounding events. • Complete hymeneal transection without a disclosure of abuse
5	Clearly Indicates Sexual Abuse	The identified signs or symptoms are extremely specific for abuse, and are not plausibly explained by medical illness or non-abusive contact.	• A young child provides specific and age inappropriate sensory details regarding the sexual abuse event. • Acute complete hymenal transection with a disclosure of sexual abuse. • Confessions, witnesses, video • Gonorrhea or chlamydia in prepubertal child > age 3 excluding non-sexual transmission • Pregnancy in a child unable to consent to intercourse.

Examples are meant to guide ratings, but are not restrictive. For example, if an expert believed that there was a non-abusive explanation for a hymenal transection (listed as an example in 4-High likelihood of sexual abuse) they would rate the case using a lower category. This communication tool should not be used in the absence of a full, expert evaluation to make or justify a sexual abuse diagnosis.

Our decision to use a 5 point scale, is admittedly, somewhat subjective. We felt like we could clearly describe 5 distinct levels of concern. The plurality of scales we reviewed used a 5-point scale, but some scales used 3, 4, or 7 categories.

Case Vignettes

We selected cases from a pre-existing, de-identified database of children who had presented for concern of sexual abuse to two Child Advocacy Centers with an annual census of more than 1000 cases with concern for any type of abuse.

We used stratified sampling to select cases with a range of abuse likelihood without selecting cases that were especially well- or poorly-suited to the scale. One author (JDM) rated all 197 sexual abuse cases in the database using the scale. Three cases from each rank were selected randomly, yielding the final group of 15 cases, from which vignettes were prepared by the authors. The author’s rankings were used only to stratify case selection to ensure a broad range of abuse likelihood, and were not considered reference determinations. Case vignettes can be seen in the online appendix

Survey Participants & Methods

Participants were recruited from the Ray E. Helfer Society, whose website describes it as “the primary sub-specialty society for physicians devoted to the problem of maltreated children.” Eligible participants indicated that they had performed or supervised at least 5 sexual abuse examinations in the past year. All Helfer Society members were sent an email invitation to participate and an individualized participation link. Invitees who did not visit the website were sent up to two reminder emails at two week intervals. No incentives or compensation were provided to participants.

Clinical vignettes were presented in random order. Participants were asked the following:

1. Among 100 children presenting in such a fashion, how many children would have experienced sexual abuse?

2. Would this case trigger the mandate to report a “reasonable concern for sexual abuse”?

3. Rate the likelihood of abuse on the given scale.

We phrased the first question to address child abuse pediatrician’s reluctance to opine a specific likelihood that a given child has been abused. This question was previously used to develop a likelihood scale for physical abuse.^2,23

Determining whether a case meets the “reasonable suspicion”^24,25 of abuse that requires a mandated report is a common professional task we expected to be familiar to the participants. Despite documented regional and individual variations in reporting behaviors,^26–28 we chose mandated reporting as our intervention outcome because it is exclusively influenced by the likelihood of abuse – as compared to other interventions such as pregnancy prophylaxis and STD testing which may be based on patient factors or the details of the alleged assault.

Participants who completed the 15 vignettes were asked for demographic information including level of training, years spent in child abuse practice, sexual abuse exams performed or supervised per month, and proportion of their professional effort dedicated to child abuse practice. Participants who completed the initial 15 items were sent an email 30 days after completion inviting them to re-rate the same 15 vignettes, and were given at least 10 days to respond.

Interrater reliability for the scale was assessed using Fleiss’ weighted kappa. Test-retest reliability is assessed using Cohen’s weighted kappa. Both kappa statistics are correlation coefficients that measure the expected agreement between two raters and range from 0 (random agreement) to 1 (perfect agreement.) The weighted version²⁹ of each statistic adjusts the statistic for more and less significant disagreements when using an ordinal rating scale. Fleiss’ kappa³⁰ extends Cohen’s kappa to the case where multiple raters have evaluated the same vignettes.

To explore the effect of participant characteristics on abuse likelihood a multivariable linear model was constructed using vignette, experience, training, and number of sexual abuse examinations per month to predict a participant’s assessment of each vignette using the proposed scale in the first wave of the experiment. This analysis was restricted to participants who completed the first wave of the experiment, and therefore had demographic information available.

Data were analyzed using R software.

Results

Among 512 members of the Ray Helfer Society invited to participate, 240 (49.6%) visited the website. 145 participants were eligible to complete the study and evaluated at least one vignette. 116 (60.4%) completed the first phase of the study. 38 of these (32.8%) participants began, and 36 completed, the second survey after at least 1 month Number of cases completed was not significantly associated with the participant’s rating in a linear regression model.

Participants were generally experienced: 78% were board certified or board eligible, and a majority completed more than 10 exams per month and had at least 10 years of experience. (Table 2).

Table 2:

Characteristics of 116 subjects who completed the first survey:

Attribute	n	%
Training
CAP Fellow	6	5.2%
CAP Board Eligible	2	1.7%
CAP Board Certified	89	76.7%
Other	18	15.5%
Experience in Sexual Abuse Exams
0-3 years	15	12.9%
4-5 years	11	9.5%
6-10 years	20	17.2%
11-15 years	23	19.8%
16-20 years	16	13.8%
More than 20 years	30	25.9%
Sexual Abuse Exams per month
0-5	18	15.5%
6-10	35	30.2%
11-15	15	12.9%
16-20	21	18.1%
21-25	10	8.6%
More than 25	16	13.8%

Test Characteristics

Participants were significantly (p < 0.001) less likely to use the extreme ratings (1 and 5) than other ratings (Figure 1). Figure 2 displays the participants’ percentage estimate of their perceived likelihood of abuse grouped by the rating on the scale. Mean perceived likelihood expressed as a percentage increased with each increment in the scale. The proportion of participants intending to report to CPS increased with higher ratings on the scale, with a large majority of participants responding that cases rated 3 or above should be reported (Figure 3).

Figure 1:

Distribution of 2394 sexual abuse rankings (166 raters x maximum of 30 ratings each)

Figure 2:

Box and whiskers plot showing likelihood of abuse by rating on the New Scale.

Figure 3.

Proportion of subjects who would make a mandated report by likelihood rating

Fleiss’ weighed kappa for the scale among the 116 participants who completed the initial survey was 0.64, indicating substantial interrater reliability as defined by Landis and Koch.³¹ Test-retest reliability among 546 participant / vignette pairs in which a participant rated the same vignette twice revealed a weighted Cohen’s kappa of 0.81, suggesting almost perfect³¹ test-retest reliability.

Inter-rater Variability

Despite the kappa statistics noted above, inter-rater variation was noted in the assessment of sexual abuse likelihood. For each of 15 vignettes, Table 3 lists the number of participants who selected each level on the scale in their first rating. Vignettes where the mean ranking was near a whole number, (e.g., vignette 11), showed a single consensus ranking. Vignettes, (e.g., # 8) where the mean ranking fell midway between two rankings tended to split the consensus between those two rankings. For 14 vignettes, at least 80% of participants ranked the case using the two ranks surrounding the mean. Because the scale has five items, 80% of the rakings fell within 20% of the scale. On the percentage likelihood of abuse scale, the difference between 10^th and 90^th centile (or the middle 80 percent) varied from 45 to 80. (Table 3)

Table 3:

Sexual abuse likelihood assigned to 15 vignettes using 2 methods: the proposed scale and percentage likelihood.

New Scale								Percentage Likelihood
Vignette	1	2	3	4	5	Mean	Within 1 of Mean	Mean	10-90th percentile
1	69	49	3	0	0	1.5	97.5 %	10 %	0 - 25 %
2	40	63	18	2	0	1.9	83.7 %	17 %	0 - 50 %
3	36	58	27	0	0	1.9	77.7 %	19 %	5 - 50 %
4	22	77	23	3	0	2.1	80.0 %	21 %	0 - 50 %
5	7	63	46	6	1	2.4	88.6 %	33 %	5 - 75 %
6	1	31	81	8	0	2.8	92.6 %	49 %	20 - 80 %
7	0	8	84	25	4	3.2	90.1 %	57 %	20 - 95 %
8	2	6	67	34	16	3.4	80.8 %	63 %	20 - 100 %
9	3	4	41	61	13	3.6	83.6 %	68 %	10 - 100 %
10	0	1	15	100	7	3.9	93.5 %	83 %	50 - 100 %
11	0	0	14	94	17	4.0	88.8 %	82 %	54 - 100 %
12	0	1	33	74	13	3.8	88.4 %	84%	50 - 100 %
13	0	0	14	91	20	4.0	88.8 %	86%	60 - 100 %
14	3	4	8	26	81	4.5	87.7 %	85%	21 - 100 %
15	0	0	0	78	53	4.4	100.0 %	92%	85 - 100 %

Note: Cases presented here in order of mean of the participants’ liklihood rating. The vignettes were presented to participants in a random order unique to each participant. The vignettes are included in the online appendix.

A multiple linear regression model did not detect a significant effect of the measured participant characteristics: experience, fellowship training, number of examinations performed per month, or percentage of professional effort in child abuse pediatrics, on the assessment of sexual abuse likelihood.

It is possible that other, unmeasured participant characteristics systematically bias the participant’s assessment of sexual abuse likelihood. To evaluate this possibility, we computed the mean ranking on the scale for each of the 116 participants who completed the first wave of the study. Because each of the participants evaluated the same 15 cases, the mean rating provides a reasonable proxy for the participant’s overall level of suspicion for sexual abuse. The distribution of the mean rating is nearly normal with a mean of 3.16. (Figure 1 in the online appendix) As discussed below, this suggests against a systemic participant bias.

Discussion

We introduce a scale to communicate perceived likelihood of child sexual abuse. The scale is intended to facilitate research regarding sexual abuse diagnosis. The scale was derived from experience with other research scales and revised in response to expert feedback. The resulting scale demonstrates substantial interrater agreement, intra-rater agreement, and is associated with intent to make a mandated report of abuse. Some of the remaining variability may reflect genuine disagreements, which are appropriately exposed by the scale.

Purpose and Scope

This scale’s primary purpose is to facilitate research related to sexual abuse evaluation. This scale is intended to express a level of concern for abuse. It is not intended and is ill suited as criteria to determine the appropriate level of concern for abuse, especially in a clinical or legal setting. While several examples are provided for each level of abuse concern, these are not intended to be comprehensive or proscriptive. The scale is silent on how multiple factors might be weighed against one another in arriving at a conclusion. We do not intend this scale to be cited as evidence that a particular case warrants any particular level of concern for sexual abuse.

Test Characteristics

The scale correlates linearly with the participants’ percentage estimates of abuse likelihood and covers a broad spectrum of perceived abuse likelihood. Whereas percentage estimates varied widely among practitioners, especially near the middle of the scale, variation in the scale is both modest and consistent across the range of likelihood. The kappas reported for this measure compare favorably with other successful research measures.³²

This theme of a general consensus, but rare unanimity, is seen in multiple studies evaluating expert assessment of sexual abuse^{16,18,19,33,34} that focused primarily on the interpretation of physical findings related to sexual abuse. These studies used ordinal scales varying from 3-7 points for the likelihood of sexual abuse based on a physical findings. Each study adjusted its metric following data collection: such as combining a 5 point scale to 3, or reporting agreement among of 11 of 12 expert reviewers as an outcome.

The scale is associated with participant’s judgments regarding mandated reporting of sexual abuse. The majority of participants indicated that they would report cases level 3 or higher. A minority of practitioners would report level 2 or even level 1 cases. Some of the diversity in reporting lower concern for abuse may be the result of different reporting standards in various jurisdictions or individual provider’s judgments about the risks or reporting or not reporting.

Our results contrast with an interview study by Jones et al,²⁶ that found wide variability in willingness to report physical abuse among primary care physicians at all levels of abuse likelihood. This contrast could be due to differences in population (child abuse pediatricians vs primary care physicians,) abuse type (physical vs sexual abuse), or study design (asking about the participants’ actual reporting vs intention to report.)

Sources of Diagnostic Variability

Diagnostic variability between participants results from two different sources. A genuine disagreement implies that the participants have different opinions about the likelihood of abuse. Alternatively, a semantic disagreement occurs when the participants interpret differently the categories by which abuse is rated. For example, two providers could genuinely disagree about whether or not a deep notch of the hymen is a concerning physical finding for sexual abuse.²⁰ Alternatively, two participants could agree completely on the proper medical and legal response to a specific clinical scenario but express a semantic disagreement as to whether the case should be designated “High Likelihood of Sexual Abuse” or “Clearly Indicates Sexual Abuse.” An ideal scale would expose genuine disagreements and minimize semantic disagreements.

Bias related to measured or unmeasured participant characteristics could be a source of genuine disagreement in the study. We inquired about participants’ training, clinical volume, and experience. Unmeasured characteristics could include the participant’s beliefs about the prevalence of sexual abuse, beliefs related to the age or gender of the perpetrator or victim, or the participant’s diagnostic decisiveness.

If a participant characteristic systemically biased a group of participant’s rankings, then the mean ranking on a common set of vignettes would differ between groups with and without the bias. One would expect a bimodal distribution of these mean rankings – formed by overlapping normal distributions of participants who did or did not have the bias.

We analyzed participant bias by looking at the mean rankings given by the participants who completed all 15 vignettes in the original survey. Multiple linear regression found no statistically significant association between the measured participant characteristics and the mean ranking given.

While we cannot directly test for bias due to unmeasured characteristics, we do note that the distribution of mean rankings is unimodal and nearly normal. While it is, admittedly, an imperfect measure the unimodal, normal distribution suggests against two or more populations of participants with systemically different views on sexual abuse diagnosis.

Strengths and Limitations

Strengths of this study include the use of randomly selected, actual cases to create the vignettes. The study also recruited a broad sample of experienced sexual abuse examiners.

Our study has three significant limitations; each of which flows from our design as an anonymous online survey. We are unable to compute a meaningful response rate. We are unable to clarify discordant responses that may indicate some difficulties with data entry. Lastly, we do not know if responses to an online survey accurately model how decisions would be made in a clinical scenario.

Response Rate:

49% of invited participants visited our website, and 60% of those were eligible to participate. Because many Helfer Society members do not perform sexual abuse examinations, we cannot compute a true response rate among those eligible to participate. If those responding to the survey held more homogenous opinions regarding sexual abuse diagnosis than the population of sexual abuse examiners, our results would over-estimate participants’ inter-rater reliability.

Discordant Responses:

We identified a discordance between the percentage likelihood of abuse and the scale in a minority of cases. For example in 11 instances a participant rated a case as “Clearly Indicates Sexual Abuse,” but assigned a percentage likelihood of abuse less than 25%. A flaw in our survey software recorded “0” if the participant neglected to enter the percentage likelihood of sexual abuse. Repeating the analysis with all “0” likelihood responses excluded yielded results similar those reported above except for modest improvements in inter-rater and intra-rater reliability (Fleiss’ weighted kappa 0.66 vs 0.64 and Cohen’s kappa of 0.82 vs 0.81, respectively.)

Representativeness:

Additionally, responses to vignettes may be different from those in clinical practice for a number of reasons. Due to our purposeful sampling, the vignettes do not represent the true distribution of abuse likelihood. We chose not to include social and behavioral factors that, regrettably, have been shown to influence clinical ju dgement.³⁵ Lastly, practitioners participating in a study of interrater reliability may give more conventional answers than when facing the realities of protecting an actual child.

Conclusions

We introduce a scale for expressing a perceived likelihood of sexual abuse that shows substantial interrater and intra-rater reliability, and is associated with intention to make a mandated report. While not designed as a diagnostic guideline, this tool may improve clinical communication of an expert’s perceived likelihood of sexual abuse, and may facilitate multi-center research in the diagnosis of sexual abuse.

We believe that this novel scale could be deployed in multi-center studies of sexual abuse evaluation in service of further validation studies, including research to determine if CAP ratings on this scale are associated with real-world testing, treatment, and reporting decisions, or longitudinal outcomes (such as investigation results, or mental health outcomes.)

What’s New:

Despite the availability of several widely used scales to communicate the perceived likelihood of physical abuse, there is currently no similar scale for sexual abuse. Initial evaluation of a scale for perceived sexual abuse likelihood shows substantial inter-rater reliability and almost perfect test-retest reliability.

Abbreviations:

CAP

Child Abuse Pediatrician