Test-Retest Reliabilities of Four Tactic-first Sexual Violence History Questionnaires

Abstract

Objective:

The present study documented, compared, and contrasted the test-retest reliabilities of the victimization and perpetration forms of a Tactic-first Sexual Experiences Survey (T-SESs) and the Post-Refusal Sexual Persistence Scales (PRSPSs).

Methods:

243 Mechanical Turk workers (116 women, 124 men) completed four questionnaires in a randomized order via anonymous web survey at Time 1 and approximately one week later at Time 2.

Results:

There were consistent gender differences in test-retest estimates. When assessing a history of victimization in women, both the T-SES and the PRSPS demonstrated evidence of minimal to good reliability (κ > .61, ICC = .86–92) while for men the PRSPS (κ = .64) was more consistent than the T-SES (κ = .59). When assessing a history of perpetration, there were fewer gender differences although post-hoc analyses suggest potential gender differences in assessing substance use facilitated perpetration (κ .48–.83) but were limited by few cases. Continuous scoring approaches were the most reliable, dichotomous scores were mostly reliable, and categorical scores generally did not meet minimal acceptable standards. For the rape victimization acknowledgment items, we found strong evidence of reliability for women (κ = .89, n = 31) and suggestive evidence of reliability for men (n = 7). There were few differences in reliability between standard and extended versions of the questionnaires.

Conclusions:

All four questionnaires exhibited good evidence of one-week test-retest reliability when scored continuously. Evidence of reliability was strongest with the populations and constructs most well studied – victimization history among women and perpetration history among men.

Sexual violence is any experience of sexual contact without consent (Koss et al., 2007). The consequences of experiencing sexual violence (i.e., sexual victimization) are often severe and include poor mental and physical health as well as challenges in relationships, parenting, and employment (Martin et al., 2011). Yet, obtaining accurate assessments of sexual violence can be extremely challenging - as exemplified by a review of men’s sexual victimization, which estimated that 2–73% of American males experience victimization (Peterson et al., 2010). These broad ranges are at least partially related to differences in measurement strategy such as varying definitions and methods of assessment (Anderson et al., 2019; Bouffard & Goodson, 2017). The inability to estimate prevalence rates hinders understanding of the scope of violence and the efficacy of prevention efforts, especially when cultural stigma already suppresses reporting and recognition of sexual violence as a serious problem (Wolitzky-Taylor et al., 2010). The goal of this study was to examine the one-week test-retest reliabilities of experimental questionnaires that use a tactic-first structure that may improve the identification of sexual violence: a tactic-first Sexual Experiences Survey – Short Form (T-SES) and the Post-Refusal Sexual Persistence Scales (PRSPSs).

Tools for Assessing Sexual Violence

The most common tools used to assess sexual violence in North American have been the different versions of the Sexual Experiences Survey (SES: Koss & Oros, 1982; Koss et al., 1987). Although the original SES had good psychometric properties it has since been revised substantially (Koss et al., 2007). These revisions have resulted in separate measures of victimization and perpetration experiences and our understanding of the psychometric properties of these questionnaires is still developing. Most researchers using the SES-Short Form Perpetration (the 2007 revised SES) modify it in some manner suggesting that even after these revisions, the SES-Short Form Perpetration does not fully meet researchers’ needs (Anderson et al., 2019). An important area for research is how the variations and modifications of the SES, such as those tested here, may impact the psychometric properties of the questionnaire in addition to the changes in prevalence rates.

One modification is to use tactic-first items. On the SES, each item begins with a description of a sexual behavior followed by a list of tactics that were used to coerce that specific sexual behavior. In tactic-first modifications, the order of these clauses is swapped; each item is oriented to the tactic followed by sub-items linking the tactic to a sexual behavior, see Figure 1. This simple change has dramatic outcomes; tactic-first versions of the SES record higher prevalence rates in both cross-sectional and experimental work (Abbey et al., 2005; Anderson et al., 2019; Schuster et al., 2020) and have initial evidence of convergent validity (Abbey, et al., 2019; Abbey et al., 2006). It is unclear why tactic-first versions increase prevalence rates, one very simple explanation is that the items are simply easier to read and understand. Another is that tactics are more effective memory retrieval cues because they have a stronger memory trace. In other words, coercive tactics are a more unique experience than sexual behavior; even if a person has experienced a large number of coercive sexual assaults they have likely experienced consensual sex more often given that the average American adult has consensual sex approximately once per week starting in the late teen years (Herbenick et al., 2010; Reece et al., 2010). Crime statistics suggest that while repeated rape is too common, it rarely occurs as often as once per week (Daigle et al., 2008). The more common an experience, the less specific details are remembered. Thus, tactics may have a more distinctive memory trace.

Figure 1. Example Items from the Questionnaires.

Note. Corresponding content is highlighted in gray on the Sexual Experiences Surveys. The PRSPS items are given in a randomized order.

In considering perpetration behavior, a focus on tactics is also more conceptually clear. Although a perpetrator may initiate sexual violence with a specific sexual outcome in mind, the tactics are the behavior the perpetrator has complete control of. The sexual outcome is influenced by both the perpetrator and the putative victim/survivor’s behavior. Indeed, this is the entire logic of feminist self-defense interventions, women cannot control perpetrator’s behavior but they can fight back (and bystanders can intervene: Orchowski et al., 2018). In other words, focusing on tactics focuses on the means rather than the ends of sexual violence, and on the target of any perpetration prevention intervention: the behavior of perpetrators.

The Post-Refusal Sexual Persistence Scales (the PRSPS: Struckman-Johnson et al., 2019) have always adopted a tactic-focused measurement strategy and were developed to capture gender differences in rape (Struckman-Johnson et al., 2003). Partially because of the tactic-focused design, the description of sexual behaviors is minimized in comparison to the SES and structure of the PRSPSs is simpler. Finally, one study suggests the PRSPS better fits the theoretical model of sexual perpetration (Testa et al., 2015).

Measurement Models for Sexual Violence and Reliability

In a latent measurement model, a questionnaire is developed wherein each item is presumed to represent a single aspect of the entire possible universe of items that sample the unobservable, underlying construct. In contrast, the measurement of sexual violence is the assessment of a history of behaviors or experiences. Thus, assessment of sexual violence history is more similar to checklists of events rather than each item representing some portion of an underlying, latent construct. Each experience or behavior of sexual violence is not necessarily related and may have occurred in different incidents, perpetrated by different people, at different ages, et cetera rather than being caused by a single underlying trait or characteristic. Although perpetration may stem from an underlying shared trait, this is an empirical question yet to be answered. However, experiences of sexual victimization are not presumed to be caused by some underlying latent factor (Koss et al., 2007). To do so would be to suggest that there is some underlying trait within victims/survivors that facilitates their victimization, rather than their victimization being caused by the behavior of another person.

An induced or formative measurement model is more appropriate for assessing sexual victimization (and potentially perpetration: Diamantopoulos et al., 2008; Hulme, 2007; Koss et al., 2007). In formative measurement models, test-retest reliability is considered the most important form of reliability whereas internal consistency reliability (e.g. Cronbach’s alpha) is appropriate for latent models (Diamantopoulos et al., 2008). Test-retest reliability estimates the stability of scores over time; in other words, are participant’s responses stable from one week to another? In the case of sexual violence, test-retest reliability provides confidence in labeling a person’s behavior or experience as rape, a serious crime; thus our focus on test-retest reliability.

Prior Estimates of Reliability – Victimization

Research suggests poorer reliability for men than women on the SES-Short Form Victimization (the revised SES for victimization) in assessing a history of victimization; with nearly one-third of cases classified differently across time for men (men: Anderson et al., 2018; women: Johnson et al., 2017). Only one study has reported kappa, which corrects for chance agreement and most estimates were < .6, the recommended cut-off for reliability (Littleton et al., 2018). Further, the worst estimates were for categorical scores (e.g., severity scores). Thus, the test-retest reliability of the revised SES for victimization may be poor, particularly for categorical scores and for men.

Embedded at the end of the SES and the subsequent revisions is the SES acknowledgment item, “have you ever been raped?” (Koss et al., 2007; Koss & Oros, 1982) This item, in combination with responses to the behaviorally specific questions, is used to categorize those who experience rape as either acknowledged, i.e., using the label rape, or unacknowledged, using an alternative and minimizing term to label their experience, such as “miscommunication” (Littleton et al., 2007). Lack of acknowledgment is common and in fact, the norm (Wilson & Miller, 2016); thus, the importance of using behaviorally specific items to detect rape. Lack of acknowledgment likely stems from internalized stigma around rape and the likelihood of negative reactions when disclosing (Ullman et al., 2007). An entire literature on the relationship between rape acknowledgment, clinical outcomes, and victimization risk has developed underscoring the importance of this construct; yet, we were unable to find any estimates of test-retest reliability of the acknowledgment item.

Prior Estimates of Reliability – Perpetration

Prior estimates of test-retest reliability for the SES-SFP (the revised SES for perpetration) have been more promising but are still quite limited. Two small studies (N ≈ 70) have found agreement between perpetration categories ranging from 82–91% over a 2-week interval (Anderson et al., 2017; Johnson et al., 2017) and a kappa of .61 (Anderson et al., 2017). But, Buday & Peterson (2015) found that when interviewing women who completed these items many women were endorsing items with victimization experiences in mind.

Current Study

The goal of the current study was to document and compare the test-retest reliabilities for two tactic-first measures of sexual violence: a T-SES and the PRSPS. We particularly wanted go analyze the data in a way that the estimates could be compared to prior literature. We chose to focus on one-week test-retest reliability as this time-frame is long enough to reduce practice effects but short enough that new episodes of sexual violence are unlikely. Thus, we compared the test-retest scores for the PRSPSs to the T-SESs, a modified version of the SES and we followed the analytic plans of Anderson et al., (2017, 2018) and Johnson et al. (2017). We also report test-retest estimates for the acknowledgment item. Finally, we also report the reliabilities of the extended versions of these questionnaires, as the extended versions include new items designed to expand the literature on sexual violence (Koss et al., 2007); yet, questionnaire length can induce participant fatigue that can hinder reliability.

Hypotheses (H)/Research Questions (RQ):

1. We expected reliability estimates to be similar if not better than in previous research, given stronger memory trace for tactics.

2. We hypothesized that the PRSPSs would have better evidence of reliability than the T-SESs given the simpler structure of the PRSPSs.

3. Given the lack of any prior research, we made no hypothesis regarding the reliability of the rape acknowledgment item.

4. We expected to find consistent gender differences, with victimization estimates being more reliable for women compared to men, and perpetration estimates more reliable for men compared to women.

5. We made no hypotheses about the reliability of the extended versions of the questionnaires compared to the standard versions.

Methods

Participants

Participants were drawn from 466 Amazon Mechanical Turk Workers who consented and opened the Time 1 survey. Only North American, adult, Workers with a high quality and completion ratings from prior tasks (>90%) were eligible to participate. The survey was advertised as “Questionnaires about Sexual Behavior 1 – Kent State University.” Time 1 participants were invited to participate in Time 2 one week later; 277 participants consented and opened the Time 2 survey. 274 of these participants were able to be matched with their Time 1 data using their Amazon MTurk IDs.

Analytic Sample.

Of these 274 participants, 46.7% were women (n = 128), 51.5% were men (n = 141) and 1.8% identified as a gender minority of some type (n = 5). Participants were mostly heterosexual (n = 235 − 85.8%), some were bisexual (n = 28 − 10.2%), gay (n = 10 − 3.6%), and one did not identify as any of the aforementioned identities (n = 1 − 0.4%). Participants were mostly Caucasian (85.0%, n = 233); 24 identified as African American (8.8%), 20 as Asian American (7.3%), and 2 as Native American (0.7%). A small number (8.0% − n = 22) identified as Hispanic or Latino/a. The average age of participants was 32.76 (SD = 7.22), range = 20 – 68. In comparison to those who did not participate in Time 2, the analytic sample was more likely to identify racially as White (69.8% vs 85.0%, χ²(1) = 15.68, p < .001), more likely to report victimization (Time 1 PRSPS, χ²(1) = 5.348, p = .021), and more likely to report perpetration (Time 1 T-SES, χ²(1) = 9.511, p = .002).

Procedures

The following procedures were approved by the Institutional Review Board of Kent State University. Data were collected in April 2018 using Qualtrics to administer the online questionnaires. After providing informed consent, participants completed the six study questionnaires (four sexual violence questionnaires, a discriminant validity questionnaire, and a demographics form) in a randomized order. After six days, participants were emailed an invitation to complete Time 2. Participants were sent an additional two reminder emails over the course of the following week and then survey access was disabled. The mean number of days between Time 1 and Time 2 was 8.66 days (SD = 2.5), median 8, and the mode was 7 days.

Measures

The Tactic-First Sexual Experiences Survey.

The T-SES used in this study is a modification of the tactic-first SES used by Abbey et al., (2005). All T-SES items are compound items beginning with a stem that describes a tactic followed by seven sub-items that represent the sexual behavior coerced: sexual touching, attempted oral sex, oral sex, attempted anal sex, anal sex, attempted vaginal sex, vaginal sex. This structure is an inversion designed to mirror the revised SES (Koss et al., 2007). The nine tactic stem items represent the following: verbal pressure (2), verbal criticism (1), alcohol/drug incapacitation (3–4), threats of physical force (1), use of physical force (1), and multiple perpetrator attacks (1). The acknowledgment item, “have you ever been raped?” was administered last. Crossing the nine tactic items by the seven sexual behavior sub-items resulted in a total of 63 items on the victimization version and 70 items on the perpetration version as the perpetration version contains an additional alcohol/drug facilitation item.

Changes made to the T-SESs.

We made a number of modifications to the questionnaire as published in 2005, reflecting advances in the science of sexual violence measurement since the idea of a tactic-first SES was first introduced (Abbey et al., 2005). Thus, the questionnaire used in this study is not a standard questionnaire but an experimental tool designed to help investigate questions related to measurement strategy and stimulate further research. We use the term T-SES not to legitimize our questionnaire as an official variant of the SES, but rather for simplicity.

We dropped mention of the gender of the perpetrator consistent with Anthony & Cook (2012)’s findings that gender-neutral instructions are more inclusive, less heteronormative, and do not appear to substantially alter psychometric properties. We separated the threat of force and use of physical force content into two items, consistent with Koss et al., (2007) and with Anderson et al., (2017)’s suggestion that a focus on specific tactics, rather than combining tactics, may improve validity and highlight potential intervention targets. We also added three items bringing the total number of items to 63 (victimization) and 70 (perpetration), respectively. Added items were drawn from the long form of the revised SES published by Koss et al., (2007). Two items were added to both forms - one describing encouraging or pressuring someone to consume substances and one regarding multiple perpetrator attacks, in order to stimulate further research on these tactics. The perpetration form included one item regarding giving someone substances without their knowledge; we only included this on the perpetration form given that this question is difficult to answer from the victimization perspective and may be better captured with other items. The T-SES items were given in a hierarchical order per Koss et al., (2007), although the sexual behavior stems were randomized within each tactic item given suggestions that randomization of items may improve disclosure rates (Dietz & Jasinski, 2007).

The Post-Refusal Sexual Persistence Scales (PRSPS: Struckman-Johnson et al., 2003).

The first iteration of the PRSPS contained 19 items; each item lists a single tactic that was used to coerce any sexual behavior such as, “threatened to break up with you.” Sexual behavior is defined broadly in the instructions as “genital touching, oral sex, or intercourse”; in this study we expanded this instruction set by adding kissing, fondling, and anal sex consistent with Strang, et al., (2013). The types of tactics assessed by the PRSPSs include enticement, verbal coercion, misuse of authority, alcohol/drugs, and physical force; thus, the PRSPSs assess two categories of tactics not included on the revised SES (enticement and misuse of authority). PRSPS items are administered in a randomized order. Construct validity for both victimization and perpetration forms were demonstrated in the original study by eliciting written descriptions of the sexual violence incidents corresponding to endorsed items (Struckman-Johnson et al., 2003).

Changes made to the PRSPSs.

We added items to equalize content with the T-SESs to control for variance introduced by content differences as prior research suggests this can be a large source of discrepancy between the PRSPS and the SES (Strang et al., 2013). Thus, in this study, the victimization version contained 24 items while the perpetration version contained 28 items. The items added were for the following content: verbal coercion (3), threats of physical harm (1) and multiple perpetrator attacks (1) for a total of 24 items. Given Strang and Peterson’s (2017) findings that perpetrators sometimes skip endorsing items that contain long lists of behaviors unless they can endorse every behavior, we separated some content into multiple perpetration items. For example, “took advantage of the fact that you were drunk or high” may adequately capture a victimization experience but could correspond to multiple perpetration behaviors including giving someone alcohol without their knowledge, verbally pressuring someone who is intoxicated, et cetera. Materials are available on osf.io. Finally, the version of the PRSPS we tested is very similar to revisions made to the PRSPS in Strang et al (2013) that they refer to as the Sexual Strategies Scale.

Traditional vs. extended questionnaires.

Although participants were presented with one questionnaire, we scored two “versions” of each questionnaire. The first version was scored with only the traditional items that appear on each questionnaire in order to provide some comparison between this study and prior estimates of reliability. These traditionally scored questionnaires are referred to simply by the name introduced in the Methods (e.g., the T-SES and PRSPS) in following analyses. The second scoring version included the additional items added for this study; these questionnaires are referred to extended versions (e.g., the T-SES extended).

Scoring.

Each item was administered with the response scale, “how many times? 0, 1, 2–5, 6–9, 10?” We tested dichotomous, categorical, and continuous scores consistent with prior research (Davis et al., 2014; Anderson et al., 2017, 2018a). We coded responses of “1” or greater on any item as endorsement of that item. Categorical scores, sometimes called severity or ordinal scores, reflect the most severe tactic endorsed by a participant: enticement (PRSPS only), misuse of authority (PRSPS only), verbal coercion, substance intoxication, physical threats/harm, multiple perpetrators. For continuous scores responses were added to a total score.

Analytic Plan and Power

We based our data analytic strategy on the methods used in prior literature to maximize comparability between our results and prior research. To that end, we report percent agreement, kappa, and correlations. Kappa of .80 is considered the cut-off for clinical constructs (strong agreement) with .6 considered a minimum (McHugh, 2012). Intraclass correlations were selected given that the ICC assumes that the data come from the same group or construct whereas Pearson’s r or Spearman’s rho are ideal for measuring the degree of association between two different constructs (Koo & Li, 2016).

In prior research the number of women with positive cases of victimization have ranged from 174 – 189, with correlations between scores in the r = .52 range (Littleton et al., 2019; Johnson et al., 2017). Sample sizes for men have historically been much smaller; ranging from 11–21 positive cases of victimization with correlations in the range of r = .04 – .53. For assessing perpetration in men, the number of positive cases has ranged from 11–17 with correlations in the r = .57 – .69 range. Research on the critical point of stability for correlations in the range of .5 suggests that a sample size of 34–68 would be adequate (Schönbrodt and Perugini, 2013). Our sample of 69+ women and 47+ men with victimization histories suggest correlational analyses may be appropriately powered. For perpetration, our sample of 24+ women and 40+ men with histories of perpetration may be under-powered; yet are 2–3x larger than prior research. We report 95% confidence intervals with 1000 bootstrapped samples to further contextualize our effect sizes. However, we recommend future research investigate the use of more complex models that are specially adapted for over-dispersed and count-type datasets.

Data Quality and Cleaning

We required MTurk Workers at Time 1 and Time 2 to be highly rated and have completed >90% of their previous tasks. Of the Workers who opted to participate in Time 2, the vast majority provided matching IDs (98.9%, unlike prior college samples: Anderson et al., 2017; 2018). We excluded 31 participants who did not complete at least 80% of either set of questionnaires so that we would have adequate data between questionnaires to compare. Most who were excluded for missing data were the same 24/25 participants who did not pass our two attention check items “please select 2–5 for this question”. Given mixed findings on attention checks questions as an indicator of data quality (Hauser and Schwarz, 2016), and our other measures of data quality we did not further restrict our sample. Skewness and kurtosis were within normal limits (+/− 3.0) for all variables except continuous variable which was kurtotic for victimization and both skewed and kurtotic for perpetration, thus we report Spearman’s rho.

Results

Victimization

Descriptive Findings.

On the T-SES, 59.5% of women (69/116) and 38.0% of men (46/121) reported a history of some type of sexual victimization on the T-SES, whereas on the PRSPS, 69.0% of women (80/116), and 50.4% of men (61/121) reported victimization. Table 1 summarizes the number of positive cases (e.g., prevalence rates) across the questionnaires.

Table 1.

Prevalence Rates by Questionnaire and Gender in this Sample

Questionnaire	% Women	% Men
Victimization
Tactic-SES	59.5	38.0
Tactic-SES extended	61.2	39.7
PRSPS-V	69.0	50.4
PRSPS-V extended	69.0	50.4
Perpetration
Tactic-SES	20.7	32.3
Tactic-SES extended	23.1	39.5
PRSPS-P	29.3	45.2
PRSPS-P extended	31.6	46.8

Note. SES = Sexual Experiences Survey, PRSPS = Post-Refusal Sexual Persistence Scale, V = victimization, P = perpetration

Dichotomous scoring.

For the T-SES, agreement was substantial for women (κ = .70) but only moderate for men (κ = .59), not meeting the minimal cut-off of .6, see Table 2. Reliability for the PRSPS was in the substantial range (κ = .61 –.64) for women and men.

Table 2.

Summary of Test-Retest Reliability Estimates by Primary Questionnaire, Gender, and Scoring Type with Effect Sizes and 95% Confidence Intervals (n = 116 women, 107 men)

	% agreement T-SES, PRSPS	K [CI] or WK [CI] T-SES/ PRSPS	Effect size T-SES/ PRSPS	Spearman’s rho T-SES/ PRSPS	ICC T-SES/ PRSPS	Minimum met?
	Victimization Findings
Women
Dichotomous	85.3%, 83.6%	.70[.56, .83]/ .61[.44, .75]	.70/ .62	—	—	Yes
Categorical	71.6%, 63.8%	.71[.61, .81]/ .65[.55, .76]	.58/ .52	—	—	No
Continuous	—	—	—	.84 [.76, .91]/ .77 [.65, .87]	.92 [.88, .94]/ .86 [.80, .90]	Yes
Men
Dichotomous	81.0%, 81.8%	.59[.43, .74]/ .64 [.50, .77]	.59/ .64	—	—	Yes - PRSPS
Categorical	71.9%,63.6%	.60[.46, .73]/ .56 [.43, .68]	.49/ .45	—	—	No
Continuous	—	—	—	.65 [.48, .79]/ .66 [.52, .79]	.79 [.70, .85]/ 79 [.70, .85]	Yes
	Perpetration Findings
Women
Dichotomous	87.1%, 87.1%	.60[.39, .78]/ .68[.53, .82]	.60/ .68	—	—	Yes
Categorical	87.1%, 87.1%	.60[.48, .85]/ .65[.51, .80]	.60/ .68	—	—	Yes
Continuous	—	—	—	.67 [.48, .83]/ .73 [.59, .85]	.92 [.89, .95]/ .83 [.76, .89]	Yes
Men
Dichotomous	82.3%, 83.1%	.60[.45, .75]/ .65[.52, .79]	.61/ .66	—	—	Yes
Categorical	75.8%, 71.8%	.60[.47, 74]/ .62 [.51, 74]	.53/ .54	—	—	No
Continuous	—	—	—	.68 [.53, .81]/ .69 [.55, .81]	.87 [.81, .91]/ .86 [.81, .90]	Yes

Note. T-SES = Tactic-First sexual Experiences Survey. PRSPS = Post-Refusal Sexual Persistence Scale. Phi effect sizes were used for dichotomous scores and Cramer’s V for categorical scores. Chi-square (dichotomous scores) and likelihood ratios (categorical scores) were used to assess statistical significance of percent agreement, all analyses were significant, p < .05. Weighted kappa is reported for categorical scores. ICC’s of .7 and above was deemed acceptable based on prior conventions (Post, 2016).

Categorical scoring.

Both questionnaires were borderline substantially reliable for women, wκ = .71 – .65 with smaller effect sizes (.52 – .58) and broad confidence intervals. Neither met minimum reliability thresholds for men (wκ ≤.6 with effect sizes < .6).

Continuous scores.

Reliability was good for women and men on both the T-SES and the PRSPS. For women, the T-SES (ICC = .92) was significantly more reliable than the PRSPS (ICC = .86), z = 2.11, p = 0.03. And both questionnaires were more reliable for women (.92 − .86) than for men (.79), z ≥ 2.13, p < .03.

Rape Victimization Acknowledgment item.

Of the 31 women who responded “yes” at Time 1, 29 responded “yes” at Time 2 (93.5%), κ = .89 (strong). Of the seven men who responded “yes” at Time 1, five responded “yes” at Time 2 (71.4%), κ = .56 (weak).

Overall summary of reliability for victimization questionnaires.

For women, both questionnaires met suggested thresholds for dichotomous and continuous scoring. For men only both questionnaires were reliable when scored continuously. Neither questionnaire was sufficiently reliable when score categorically for either gender. Estimates were in general stronger for women than men.

Perpetration.

Descriptive Results.

On the T-SES for perpetration, 20.7% of women (24/116) and 32.3% of men (40/124) reported a history of sexual perpetration. On the PRSPS, 29.3% of women (34/116) and 45.2% of men (56/124) reported sexual perpetration.

Dichotomous Scoring.

Both questionnaires were reliable when scored dichotomously, for women, κ = .60–.68 and men, κ = .60–.65 see Table 2.

Categorical Scoring.

Both questionnaires were sufficiently reliable for women when scored categorically (κ = .60–.65) while for men effect size estimate suggest it may not be (<.6).

Continuous Scores.

Both questionnaires were reliable for men and women. For women, the T-SES (ICC = .92) was more reliable than the PRSPS (ICC = .83), z = 2.86, p = 0.004 while they were nearly equivalent for men (.86–.87).

Overall summary of reliability of perpetration questionnaires.

Both questionnaires were acceptable when scored dichotomously or continuously for either gender. Neither questionnaire was acceptable for men when scored categorically.

Standard vs. extended versions.

Victimization.

Victimization prevalence rates on the extended versions were equivalent on the PRSPS and higher than the standard T-SES (women: 59.5 vs. 61.2/ men 38.0 vs. 39.7). Although there were some numerical differences (see Table 3), there were no interpretative differences in reliability estimates between standard vs. extended versions of the T-SES and the PRSPS in assessing victimization.

Table 3.

Reliability of the Extended Questionnaires

	% agreement T-SES ext/ PRSPS ext	κ or WK T-SES ext/ PRSPS ext	Effect Size T-SES ext/ PRSPS ext	Spearman’s rho T-SES/ PRSPS	ICC T-SES ext/ PRSPS ext
	Victimization Findings
Women
Dichotomous	85.3%, 83.6%	.70[.57, .83]/ .61[.44, .77]	.71, .62	—	—
Categorical	69.8%, 59.5%	.71 [.62, .81]/ .65 [.55, .75]	.60, .51	—	—
Continuous	—	—	—	.85[.76, .91]/ .77 [.65, .86]	.91 [.88, .94]/ .84 [.77, .89]
Men
Dichotomous	80.2%, 81.8%	.58[.41, .73]/ .64[.50, .77]	.58, .64	—	—
Categorical	66.9%, 61.2%	.59[.46, .72]/ .56[.43, .68]	.45, .46	—	—
Continuous	—	—	—	.67[.52, .79]/ .65[.49, .78]	.82 [.74, .87], .79 [.70, .86]
	Perpetration Findings
Women
Dichotomous	83.8%, 85.5%	.54[.35, .70], /.66[.49, .80]	.54/ .66	—	—
Categorical	82.9%, 78.6%	.61[.43, .81], /.66[.53, .79]	.66/ .56	—	—
Continuous	—	—	—	.61[.39, .77]/ .74[.59, .84]	.94 [.91, .96]/ .84 [.76, .89]
Men
Dichotomous	83.7%, 84.7%	.70[.57, .82]/ .69[.56, .81]	.70/ .880	—	—
Categorical	78.2%, 69.4%	.70[.58, .82]/ .62[.51, .74]	.56/ .50	—	—
Continuous	—	—	—	.73[,60, .84]/ .72[.60, .84]	.86 [.79, .90]/ .86 [.80, .90]

Note. T-SES = Tactic-First sexual Experiences Survey. PRSPS = Post-Refusal Sexual Persistence Scale. Phi effect sizes were used for dichotomous scores and Cramer’s V for categorical scores. Weighted kappa is reported for categorical scores. ICC’s of .7 and above was deemed acceptable based on prior conventions (Post, 2016).

Perpetration.

Perpetration prevalence rates on the extended versions were higher than the standard versions although the differences were small, up to 2.4 percentage points for women and up to 7.2 for men. Although there were some numerical differences (see Table 3), the extended versions of the T-SES and PRSPS generally performed very similarly. Exceptions were all for women; dichotomous scores on the T-SES did not meet standards nor did PRPS categorical scores.

Post-hoc analyses.

Given the poor kappas for categorical scores, we also computed follow-up analyses to evaluate reliability for each tactic individually. Categorical scores assign each person a category corresponding to the most severe tactic they experienced; thus, error variance is compounded and variably distributed across participants who range in the number of tactics they experienced. Given that the number of cases per tactic was sometimes very small by gender, ranging from 7 – 33, we report findings for the entire sample using the standard questionnaires.

Victimization.

Kappas on the T-SES tactic scores (verbal, substance use, physical force, multiple perpetrator) ranged from .65–.66. For the PRSPS, kappas ranged from .55 – .64. For the PRSPS, kappas for verbal coercion (.57) misuse of authority (.58) and multiple perpetrator (.55) did not meet standards. We repeated these analyses by gender; kappas more than .10 worse for men were for substance use and physical force on both questionnaires.

Perpetration.

Kappas on the T-SES ranged from .60 to .7. Estimates for the PRSPS ranged from .43 – .65 with enticement (.56), misuse of authority (.49), substance use facilitation (.57), and multiple perpetrator attacks (.43) failing to meet minimum thresholds. When repeating these analyses by gender, T-SES substance use facilitation improved for women (.83 from .60) and worsened for men (.48). Similarly, on the PRPS, substance use was more reliable for women (.67 from .57) when analyzed separately.

Discussion

One barrier to sexual violence research has been the lack of precision in the measurement of violence; and specific to the assessment of sexual violence, the lack of test-retest reliability data. Our findings document initial evidence of reliability for a set of tactic-first questionnaires. We found that reliability estimates reflect the biases of the current literature – reliability estimates for victimization for men were poorer as were perpetration estimates for women.

Victimization Findings

Our findings highlight the complexity of assessing sexual victimization. Generally, there were few differences between the T-SES and the PRSPS (H2) and gender and scoring approach appears to be more salient factors in explaining differences in reliability than questionnaire. The only approach to assessing victimization that was reliably consistent across genders were continuous scores (H4). Indeed, although we hypothesized that the reliability estimates reported in this study would be stronger than those reported in previous literature (and they generally were: H1), we were surprised that our estimates still failed to meet minimum standards in many cases. This suggests that percent agreement as reported in prior research (Johnson et al., 2017; Anderson et al., 2017; Anderson et al., 2018) is a misleading assessment of reliability for sexual violence questionnaires. The poor reliability of categorical scores is consistent with Littleton et al., (2018)’s findings, the one prior study that reported kappa. We suggest that reliability for categorical scores is worse than dichotomous or continuous scores because categorical scores compound measurement error by coding only the most severe experience of a variable range of experiences. We recommend researchers avoid this scoring method.

The reliability of the T-SES rape acknowledgment item was good for women (H3). In spite of the growing literature on rape acknowledgment and clinical outcomes (Wilson & Miller, 2016), little has been reported on the psychometric properties of this item. Reliability of this item was also good for men but given the sample size (n = 7), we consider this finding very preliminary. In addition, using only the acknowledgment item would under-estimate cases by orders of magnitude, in this study, reducing the identified cases of sexual victimization by at least half for women (26.5 % vs. 59 – 69%) and even more for men (5.8% vs. 33 – 50%).

Perpetration and post-hoc analysis findings

Perpetration findings mirrored victimization findings – although gender differences were less pronounced. We were surprised there was not stronger evidence of reliability for the PRSPS given Testa’s findings on structure and that the PRSPS was designed with gender differences in mind (H2: Struckman-Johnson al., 2003). In general, there were few differences between the questionnaires especially for perpetration. Post-hoc analyses suggested which tactics are least reliable and highlighted potential gender differences. This suggests that future research concentrate on better understanding the victimization experience via verbal coercion, misuse of authority, and multiple perpetrators. Notably, multiple perpetrator attacks are less common, and our reliability estimates may reflect poor power due to the small number of cases overall and especially when analyzed by gender. We were surprised by how poor reliability estimates were for perpetration via substance use facilitation on the T-SES, especially in comparison to the same content on the PRSPS. However, Littleton et al., (2018) also found low kappas estimates for substance use victimization. This finding may reflect the uncertainty introduced by alcohol in sexual situations – while initial consumption may be consensual or planned, continued consumption introduces uncertainties that may be difficult to assess afterwards given the initial context and pharmacological effects of alcohol. Secondly, these gender differences may reflect false positive rates for women on perpetration measures (Buday and Peterson, 2015). Poor reliability, specific to women reporting on perpetration via substance use may reflect that some of these women were actually thinking of victimization experiences (H4). We have strong reservations about using these questionnaires to assess perpetration in women without further research. There were also few differences between the standard and extended versions of the questionnaires which is assuring for users who wish to adding items from the long form as we did (H5).

Limitations

While this study represents one of the larger samples to date our sample size overall was still small and subsequently, contained only a small number of positive cases to analyze. We did not have an adequate sample size to test for effects related to sexual or gender minority status. This is an important point as sexual and gender minority people experience sexual violence at a much higher rate compared to heterosexual people (Rothman, Exner, & Baughman, 2011; James et al., 2016), and currently available measures may not fully capture their experiences (Dyar at el., 2019). It is imperative that this type of work continue with a range of populations – our findings are specific to American adults on MTurk. MTurk participants tend to be more educated, less religious, less likely to be employed, less extraverted and have lower self-esteem (Goodman et al., 2013). The MTurk population also tends to learn and change rapidly; thus methods for assessing data quality are uncertain and evolving (Hauser et al., 2018; Chandler and Shapiro, 2016). We did not randomly assign participants to the standard versus extended versions of the questionnaires. Finally, some research suggests more complex data analytic approaches are needed for this type of data (Swartout et al., 2015).

Research Implications

Our findings suggest that tactic-first questionnaires increased reported prevalence rates and are at least somewhat reliable measures of sexual violence. As suggested by Buday and Peterson (2015), the gender differences found here reflect the literature – existing questionnaires contain embedded gender biases that reflect a gender stereotypic range of experiences. For example, recent research suggests that being made to penetrate may be a somewhat common form of victimization against men by women (Anderson et al., 2020). It is possible that qualitative research to develop new items is necessary.

Estimates of reliability in this study were generally in the minimally acceptable range (H1). This is notable given the larger number of positive cases (24–73) in our study compared to prior work (Anderson et al., 2017; Anderson et al., 2018; Johnson et al., 2017). Yet, larger samples may be required. It is also possible that strong reliability for this type of behavioral measure is just more difficult to achieve for a number of reasons. Sexual victimization is not a latent construct and the theory of measurement for induced models is still being developed. Another complicating factor is that the time-period, e.g., “since you were 14” often spans years if not decades. Finally, some of the behaviors assessed may be highly context dependent. Using verbal means to ‘repeatedly ask for sex” may be interpreted quite differently across participants and what may be coercive in one setting may be consensual but unwanted in another, belying simple behavioral descriptions.

Clinical and Policy Implications

This study provides valuable data for individuals measuring sexual violence. Our findings underscore the value of kappa instead of percent agreement as assessments of reliability. The present findings also underscore the need for further research on the use of tactic-first questionnaires and research designs that cut against the stereotypical, yet also very common paradigm of men as perpetrators/women as victims. Though research on this stereotypical dyad is necessary and important, our results underscore the frequency of men’s victimization and women’s perpetration and current science’s limitation in understanding these phenomena. The frequency of men’s victimization and women’s perpetration suggests these phenomena also deserve much greater clinical attention – including screening and direct therapeutic intervention. Finally, although we tested experimental questionnaires modified to reflect recent advances in science, these tools still do not capture all salient factors. For example, the SES method for assessing the relationship to the perpetrator of violence is inconsistent with some health surveillance systems, such as those used by the World Health Organization (Garcia-Moreno et al., 2005). Another area for future research is examining the taxonomy of violence and how to create questionnaires that adequately represent these taxonomies (Bagwell-Gray, 2019; Cook & Parrott, 2009).

Conclusions

This study demonstrated adequate test-retest reliability for four tactic-first sexual violence questionnaires. The reliability of these tactic-first questionnaires was comparable to the traditional sexual behavior-first versions and better in some instances. Reliability was strongest for the questionnaires and populations that have historically been has studied the most – women who experience victimization and men who perpetrate. Reliability was worse for the experiences least studied – men’s victimization and women’s perpetration. Our findings underscore the potential usefulness of tactic-first questionnaires while highlighting the need for more gender-expansive research in sexual violence.