The Development and Piloting of a Digital Checklist to Increase Access and Usage of Campus Online Sexual Violence Resources

Abstract

As colleges seek to respond to campus sexual assault, administrators are making policies, programs, and resources related to sexual violence available to students and members of the school community online. Keeping this content current and accessible can be challenging in the context of rapidly changing information and competing priorities across campuses. In response to this challenge, we developed a free, online protocol for campus leaders to quickly assess gaps in their online sexual violence resources. The “Digital Checklist” delineates action steps that campus administrators can take to determine whether information related to campus sexual assault is easily located, current, relevant, and accessible. We found that while schools made an effort to increase the availability of information online, the checklist helped identify gaps that, if remedied, would allow more students to access that information. The overall goal for the checklist is to provide schools with actionable, real-time, and trackable data regarding the utility and accessibility of online sexual violence information and resources and to complement other campus sexual assault prevention and intervention efforts.

The past decade has brought an increased focus on student rights around sexual assault prevention and services related to Title IX (U.S. Department of Education, 2015), a federal law that prohibits discrimination on the basis of sex in education programs. High-profile statements, including the “Dear Colleague” letter from Assistant Secretary for Civil Rights Russlynn Alli in 2011 (U.S. Department of Education, 2011), as well as the White House issued “Not Alone” report in April 2014 (White House Task Force to Protect Students From Sexual Assault, 2014), have led to increased attention on the clarification, enhancement, and implementation of campus-based sexual violence prevention, policy, and enforcement.

In the current digital age, members of college and university communities often seek information online related to sexual violence to access services, understand policies, and elicit help for themselves and others (Levine, 2011). Access is a function of two constructs—(1) how people are able to get to the information and (2) how they are able to understand and use this information for varying needs. This is conceptualized as digital health literacy, which is defined as the way individuals obtain, access, and use health information online (Norman & Skinner, 2006). Because college students have low levels of digital health literacy (Stellefson et al., 2011), increased attention is needed to how and where those students access health-related content online. Therefore, standardized procedures and practices that make a school’s online information and resources accessible for diverse campus community members—including students, faculty, staff, and community partners—may help ensure that members can readily find appropriate content that is current, relevant, and accessible to individuals with various needs and aligning with the Americans with Disabilities Act (ADA) recommendations (Americans with Disabilities Amendments Act of 2008, 2009). With ever-changing content and significant competing demands on schools, it is challenging to keep this content current and accessible.

The current project emerged from a larger initiative funded by the U.S. Department of Health and Human Services Office on Women’s Health to improve the response to campus sexual assault and assist postsecondary institutions in responding effectively to sexual assault on their campuses. As part of this project, a free, online tool for campus leaders was developed to assess gaps in their online sexual violence resources platform. This “Digital Checklist” delineates action steps that campus administrators can take to determine whether information related to campus sexual assault is easily located, current, relevant, and accessible (e.g., determining how “searchable” their school’s resources are through common search terms; clicking on links for current/active sites; ensuring content has sufficiently large font). Across various fields, checklists have been used to help standardize procedures, enhance thoroughness, and prevent mistakes (Gawande, 2010). Similarly, this checklist strives to provide schools with a tool to ensure online sexual violence policies, resources, and materials are accurate, comprehensive, and accessible.

The current initiative includes eight postsecondary institutions that are highly diverse with respect to geography (East Coast, South, Mid-West, West Coast), size (ranging from 1,500 to more than 40,000 students enrolled), student population (full-time, part-time, commuter), and funding source (i.e., public vs. private). As part of the larger initiative, each campus had a leadership team that included campus staff or administrators responsible for overseeing the Title IX and Sexual Violence Prevention programming. Each of these teams completed an environmental scan (a 77-item tool) at the start of the academic year. The environmental scan included questions that assessed school climate and culture by asking questions related to school-wide collaboration, data collection and evaluation, prevention and education, reporting processes, investigation, adjudication, survivor support services and accommodations, and workplace policy. Additionally, campus sexual violence prevention policies were reviewed twice a year as part of a policy scan conducted by the project team to assess adherence to federal mandates and best practices. Given this biannual policy scan, the Digital Checklist was designed to assess the online presence of such policies rather than their content.

In evaluating the first 2 years of environmental scans, all eight schools self-reported providing appropriate, timely, and relevant content online. However, on further examination of the content available online, there was recognition that while a variety of information was available about the campus, the broader environmental scan did not fully address the accessibility or relevance of campus resources for diverse populations. Thus, it became clear that checking whether information existed was not enough, consideration needed to be given to how easily information could be found and the utility of the information available for a wide range of student populations. It was also found that, consistent with research on how undergraduate students find information, students were often using internet searches, rather than navigating from the school’s homepage to find needed information (Kirkwood, 2008). This led us to experiment with some key search terms with partner schools’ names to determine what resources came up via an internet search. Ultimately, through keyword searches on popular search engines, it was ascertained that the information, while technically there, was not always accessible by the methods most used by intended audiences. Therefore, through a consensus-based, iterative process, a standardized procedure (the Digital Checklist) was developed to allow for ongoing monitoring of online content with attention to issues such as ease of searchability, accessibility, and relevance of content to diverse groups. To keep the Digital Checklist free and accessible, it was developed using an online Google document that each school could complete biannually.

This article describes the development of the Digital Checklist and pilot implementation with six campuses partners, including lessons learned and recommendations for future use and evaluation. Data from the two additional partner campuses were not included in the analysis because they did not complete the checklist during the initial wave of pilot data collection. The overall goal for the checklist is to provide schools with actionable, real-time, and trackable data regarding the utility and accessibility of online sexual violence information and resources and to complement other campus sexual assault prevention and intervention efforts.

Development of the Digital Checklist

The Digital Checklist is a series of multiple-choice, dichotomous (yes/no), and qualitative (text box) questions. The checklist also provides links to external resources, such as the ADA Best Practices for Website Accessibility document (ADA, 2007) and reading grade-level analysis programs (U.S. Department of Health & Human Services, n.d.). Partner schools signed MOUs (memoranda of understanding) to participate in the program and evaluation components of the project, and the study was approved as an exempt protocol by University of Pittsburgh Human Subjects Research Protection Office. See Table 1 for the full Digital Checklist.

Table 1.

Digital Checklist.

Question number	Section: Online presence
1	Do all webpage links to resources, policies and offices related to sexual violence work go to the correct location on your website? Please identify webpages you need to fix or update.
2	Are all sexual violence related webpages easy to read on a mobile phone or tablet? Please identify webpages you need to fix or update.
3	Are all digital resources and webpages compliant with ADA best practices? Click the box if your website is compliant with best practices. Please refer to the ADA.gov toolkit for further explanation of best practices (https://www.ada.gov/pcatoolkit/chap5toolkit.htm) • Colors and fonts can be changed in the visitor’s browser • Font types are readable (Sans Serif, Arial, Helvetica, Verdana recommended) • Font size is at least 12 or 14 pt • Images include alternative text tags or long description tags that provide text descriptions • Page avoids use of extra thin, extra bold, or extra italicized fonts • Page does not utilize “click here” for hyperlinks and instead hyperlinks full phrases that note where the link is going • Videos include closed-captions • Page has adequate color contrast ratio between text and background (https://soap.stanford.edu/tips-and-tools/tips/contrast) • Documents are posted in a text-based format like RTF or HTML in addition to PDFs, which are often not compatible with screen readers and other assistive technologies
4	Does your Title IX page list and explain confidential versus nonconfidential resources?
5	In your opinion, how clear and comprehensive is the information about confidential versus nonconfidential resources?
6	Does the Title IX policy page have a brief overview or “cheat sheet” of reporting processes and resources so that someone does not have to search through the entire policy?
7	Test the grade level of your school’s sexual violence policy using this website: https://www.online-utility.org/english/readability_test_and_improve.jsp. What is the Flesch Kincaid reading level of your sexual violence policy language?
8	Does your school’s main prevention and response page prominently link to the Title IX page and policy?
9	Do your school’s prevention and response pages list and define multiple forms of sexual and gender-based violence? Check all the terms listed below. • Sexual assault • Rape • Sexual harassment • Dating violence • Intimate partner violence • Domestic violence • Stalking
10	On your school’s main homepage, what are the first 3 results when searching the following terms? Please list them in the spaces below. • Rape • Sexual assault • Sexual harassment
10a	In your opinion, are these results the most relevant and timely resources?
11	Does your school’s main prevention and response page list information about both short term and long term support options for survivors? • Short-term (where to access a rape kit or toxicology exam, crisis hotline, police escorts) • Long-term (counseling resources, trauma-informed yoga classes, reporting, academic accommodations)
12	Are links to campus and local sexual violence resources prominently displayed on your campus safety/police page?
13	Do your campus safety alerts (e.g., emails/texts) notifying community members about sexual violence incidents include information about local and campus survivor support resources?
14	If you partner with a community-based victim advocacy program, do they have links to your campus resources prominently displayed?
15	Where does your school list or link to your Clery Act data?
16	What search terms do you enter to access your school’s resources?
16a	How long did it take you to access your school’s resources using a search engine?
17	Do you and your staff include language in your email signature about sexual violence/harassment policy and resources?
18	Do you regularly search social media (Facebook, Twitter, etc.) for related posts using “your campus name + sexual assault or harassment”?
19	Have you set up a Google news alert for “your campus name + sexual assault or harassment”?
20	What initial results come up when searching your school + typical search terms on Google? List the results below for each search term. Try this on a public computer or after clearing browser’s cache/history/cookies. • Search: [your school + rape] (#1–3) • Search: [your school + sexual assault] (#1–3) • Search: [your school + sexual harassment] (#1–3)

Note. ADA = Americans with Disabilities Act; RTF = rich text format; HTML = hypertext markup language; PDF = portable document format.

Four members of the project team were involved with the initial set of items for the checklist. Members of the project team brought expertise from clinical social work, education, campus-based sexual violence prevention, and public health. The process began by developing language universally used around these issues, then eliminating topics that were redundant with the environmental and policy scans being used. The search terms were selected based on language used within the environmental and policy scans. Next, based on the scope of expertise of the project team and in consultation with additional experts in the field, the items were narrowed down to include those most salient to issues specific to campus sexual violence. Finally, feedback and commentary from evaluation partners was sought and incorporated for the piloted version of the tool. The Digital Checklist has two broad categories: (1) Online Presence, which focuses on keyword searches on internal campus webpages, hyperlinks, adherence to ADA best practices (including reading level) and (2) Digital Engagement, which highlights engagement around social media platforms, social media monitoring, and external searches via Google.

Online Presence

Information Provision.

The checklist asked several questions related to the provision of information including presence, location, and visibility of information on the school’s website. An additional question inquired about alerts as a strategy to inform the campus community about incidents of sexual violence with information and resources.

Keyword Searches Using the Embedded Search Tool.

The Digital Checklist instructs officials at partner schools to start on their school’s main webpage and then using the embedded search function to navigate to associated internal webpages related to campus sexual assault. Search terms include “rape,” “sexual assault,” and “sexual harassment.” Respondents are then asked to assess the relevance of the resources identified through the search on a Likert-type scale of 1 to 5, with 1 being completely relevant and 5 being completely irrelevant (with relevance reflecting how appropriate resources were for students seeking information around the aforementioned topics).

Hyperlinks.

Respondents are asked to check if all of their internal homepage hyperlinks related to sexual violence polices, resources, and related campus offices are correct and if there are webpages that need to be fixed or updated.

ADA Compliance and Accessibility.

Check boxes allow Digital Checklist respondents to indicate whether their website and associated resources are aligned with federal best practices (Bureau of Internet Accessibility, 2019) as well as best practices related to accessibility from the field. Some of these best practices include the closed captioning of videos, adequately large font size (i.e., minimum of 12 point), contrast ratios, text tags, and the availability of documents in text-based formats (ADA, 2007). Compliance with website accessibility standards allows diverse members of the school community greater access to critical information.

Reading level is another important criterion for accessibility. It is important that the entire campus community understands the policies, even if they are student directed. While some may argue that a reading level of college is appropriate for a college community, policies are implemented and affect all members of the community, which includes not only students and faculty but also staff at all levels, parents, and community partners.

Digital Engagement

Social Media.

Social media searches are critical given the amount of time that college age students spend on social media platforms. A Pew study found that college-aged individuals are the highest users of social media (Smith & Anderson, 2018). Therefore, the Digital Checklist includes a section for schools to indicate whether they currently search social media (Facebook, Twitter, etc.) for student posts related to sexual assault and sexual harassment. By conducting social media searches, schools can better understand student experiences and perceptions related to sexual violence on campus. This perspective may shed light on unknown issues and offer opportunities for increased prevention and intervention.

Google Alerts.

“Google Alerts” are a valuable tool to help schools consistently monitor their online profiles (i.e., what type of information is being disseminated about the school online). Google alerts can be set to scan the internet at regular intervals looking for uses of a particular key word or key words (e.g., the name of the school) and send regular email reports with this information. The information provided can range from the school being named in a news article to a student tweeting about their experience with school resources. Campus respondents are asked to identify whether they set up Google alerts for terms such as “campus name + sexual assault or sexual harassment.”

Keyword Searches via Google.

Students consistently report that when searching for Title IX resources, sexual assault prevention programming, or sexual assault response services, they used an internet search engine, such as Google, to find the information (Kirkwood, 2008). To determine whether this was an efficacious way of accessing the school’s resources, common search terms combined with the schools’ names were put into internet search engines. As a result, it was found that the sources that schools worked so hard to cultivate online were, in fact, not consistently at the top of the search list. For example, when the name of a major university in combination with the search term “sexual assault” was googled, the first result was an article written by a student about their experience of sexual assault on that campus. Therefore, keyword searches were included in the Digital Checklist for schools to assess the accessibility of their sexual violence related resources.

Pilot Implementation and Initial Findings

Six of the eight schools engaged in the larger initiative participated in the development of the Digital Checklist presented here. The primary program leads, who took part in monthly calls and in-person initiative meetings, completed the pilot of the Digital Checklist. These campus program leads included Deans of Students, and Coordinators and Directors of Title IX compliance and campus sexual violence prevention and response services. Table 2 presents the characteristics of the pilot schools. Two broad categories, of Online Presence and Digital Engagement, are presented with information related to how schools responded within each of the corresponding components.

Table 2.

Characteristics of Participating Institutions.

School	Location	Approximate enrollment size	Carnegie classification	Private or public	NCAA division	% White students
1	Northeast	30,000	R1: Doctoral universities—Very high research activity	Public	NCAA Division I	70%
2	Northeast	1,500	M3: Master’s colleges and universities—Smaller programs	Private	NCAA Division III	62%
3	Midwest	14,000	Associate’s colleges: Mixed transfer/career and technical—High nontraditional	Public	NJCAA Division II	67%
4	Southeast	5,000	M2: Master’s colleges and universities—Medium programs	Public	NCAA Division III	68%
5	West	42,000	R1: Doctoral universities—Very high research activity	Public	NCAA Division I	26%
6	West	11,000	D/PU: Doctoral/professional universities	Private	NCAA Division I	25%

Note. NCAA = National Collegiate Athletic Association; NJCAA = National Junior College Athletic Association.

Online Presence

Information Provision.

Results demonstrated that the schools were proactive with respect to providing information through multiple channels, but some areas were stronger than others. All schools reported that their prevention and response webpages list both short- and long-term options for survivors and provide definitions of sexual assault, rape, and sexual harassment, and almost all (five out of six) provide definitions of domestic violence and stalking. Five out of six schools indicated that they send safety alerts and resources to school community members about sexual violence incidents and four out of six indicated that they also include sexual violence and sexual harassment resources and policy information in their email signatures. Four out of six schools reported that their webpages list and explain confidential versus nonconfidential resources and include brief overviews or “cheat sheets” of their Title IX policies. Last, three out of six schools indicated that campus and local sexual violence resources are prominently displayed on their campus safety/police pages.

Keyword Searches Using the Embedded Search Tool.

Each of the six schools who responded listed the top three results for keyword searches using the embedded search tool on their campus websites for the terms rape, sexual assault, and sexual harassment. Two of the six schools indicated that their search results were completely or mostly relevant to the topics being searched.

Hyperlinks.

All six of the schools that responded indicated that their links were current and up to date. Five out of six schools also indicated that their resources and webpages were easily readable on mobile phones or tablets. Four out of six schools indicated that their prevention and response page prominently linked to their Title IX policy webpage.

ADA Compliance and Accessibility.

Overall, campus websites were mostly compliant with recommendations around readable font type, adequate font size, and the utilization of full phrases for hyperlinks. However, when partner schools were asked to calculate the reading level of their policies using the Fleish Kincaid Test (HHS.gov, 2015), the policies of each of the five schools who responded to the question was above the recommended reading level of 8th grade for documents for the general public and a 12th grade reading level for readers with college degrees (HHS.gov, 2015). The average grade reading level across five schools who responded was 14.35. Additionally, partner schools indicated low levels of adherence of best practices around the ability to change colors and fonts in the visitors’ browser, images that include alternative text tags or long description tags that provide text descriptions, closed captioned videos, and adequate color contrast ratio between text and background. See Table 3 for additional school responses related to accessibility.

Table 3.

Results From Six Pilot School on Items Related to Accessibility and Key Sexual Violence Information.

Accessibility items (based on federal best practices)	Number of schools indicating “yes”
1. Colors and fonts can be changed in the visitor’s browser	1
2. Font types are readable (Sans Serif, Arial, Helvetica, Verdana recommended)	5
3. Font size is at least 12 or 14 pt	4
4. Images include alternative text tags or long description tags that provide text descriptions	2
5. Page avoids use of extra thin, extra bold, or extra italicized fonts	4
6. Page does not utilize “click here” for hyperlinks and instead hyperlinks full phrases that note where the link is going	5
7. Videos include closed-captions	2
8. Page has adequate color contrast ratio between text and background	2
9. Documents are posted in a text-based format like RTF or HTML in addition to PDFs, which are often not compatible with screen readers and other assistive technologies	4

Note. RTF = rich text format; HTML = hypertext markup language; PDF = portable document format.

Digital Engagement

Social Media.

Four out of the six schools in the pilot responded that they do not regularly search social media (Facebook, Twitter) for the aforementioned terms.

Google Alerts.

Five out of six schools in the pilot indicated that they do not currently have alerts for the aforementioned terms.

Keyword Searches.

As part of this initiative, the schools who responded listed the top three search results for their school name plus the search terms sexual assault, rape, and sexual harassment. Follow-up work is currently being done with each school to identify the steps they are taking to improve the relevance of their search results.

Discussion

The preliminary evaluation of digital information related to sexual violence on six campuses via a free Digital Checklist found substantial issues with accessibility of online content, as well as issues with lack of relevant content (e.g., links to local community resources), and lack of digital monitoring to improve relevancy of content and promote school awareness of extraneous content. When partner schools were asked to calculate the reading level of their policies, every school’s policy was well above recommended reading level of 8th grade for the general public and 12th grade for college audiences (HHS.gov, 2015). As a result, the project team is actively working with schools to determine how to make these policies more accessible by either using more plain language or creating companion documents (e.g., visual aids flow charts) to allow them to be understood widely given the legal language required in these policies.

Regarding the appropriateness and relevance of sexual violence related resources, the Digital Checklist is intended to be used as an opportunity for reflection rather than a search optimization tool. The utility of the checklist lies in empowering each campus to take regularly scheduled, in-depth explorations of their online environment and content related to sexual violence and then reflect on the findings to determine what, if any, improvements should be made. Illustrating this point is the feedback thus far received from the pilot schools. For example, one school found that their content was easily viewable on iPhones, but not Android phones and tablets and this was a priority to address with their information technology department, indicating that a future iteration of the checklist should include more specific questions related to viewability across different device brands and software platforms. Another school indicated that they would be updating all websites to ensure compliance with ADA best practices as well as developing a cheat sheet (using simplified language) for their Title IX policy.

Recommendations for Use, Strengths, and Limitations

It is important to note that this Digital Checklist is nested within a larger initiative and evaluation framework. Specifically, broader environmental scans captured many dimensions of campus environment (e.g., prevention, policy, response, adjudication), while the Digital Checklist is focused solely on the online environment. It is recommended that campus sexual violence leadership teams complete and review their Digital Checklist data biannually to facilitate informed discussions on strengthening weak or inaccessible areas. The tool is not intended to be a stand-alone intervention, but rather one tool within the campus’ broader and systematic efforts to address sexual violence.

A notable strength of the Digital Checklist is that it is free and easily accessible (campuses interested in using this tool may contact the first author for a free copy, and free download access is forthcoming on www.futureswithoutviolence.org). The checklist was designed to be easy to use, and can be filled out by a single person, with most results being easily understood for individuals with basic technological knowledge related to internet and social media use. The checklist is also adaptable based on the needs of the school, with additional questions easily added through the online form. It should be noted, though, that it is advisable for schools to keep track of the outcomes of using the checklist (e.g., addition of cheat sheets for complex policies, streamlining access, and visibility of resources for sexual assault survivors) as well as any potential campus impacts resulting from use of this tool over time. Campuses may also wish to track the number of contacts with the Title IX or other campus resources office over time, as well as how each person contacting the service found information about it, to see which digital changes seem to correspond with improved service use.

The Digital Checklist is not without its limitations. First, it functions as a digital scan rather than a digital or search optimization tool. The checklist can help schools identify, at a basic level, their digital presence and visibility. While it does include some basic resources related to accessibility standards, it does not offer technical solutions to remedy any issues located using the checklist. For this reason, this tool is recommended for use as part of a broader effort around campus-based sexual violence prevention and intervention. Second, the checklist only prompts campuses to examine the readability of the sexual assault/sexual misconduct policies. Schools were not asked to check the entirety of their sexual violence materials available online. Third, the overall utility of the checklist would benefit from increased specificity, uniformity of formatting, and clarity. These forthcoming updates are described in detail in the section on future directions. Finally, the tool has not been formally evaluated for reliability, validity, or effectiveness in prompting change on campuses. Such an evaluation was beyond the scope of the present project.

Future Directions

As this was an initial pilot of the checklist, areas for improvement have been identified. Future iterations of the checklist will include the following: First, to be more comprehensive, items allowing for a full scan of Clery compliance and adherence to federal and state mandates will be added. To be more accurate and comprehensive, checklist items referencing Clery Act data will be changed to instead reference Annual Security Reports. Second, to increase specificity, open-ended questions will be changed into discrete checklist items as well as will ask respondents to indicate the number of clicks it takes to reach the school’s sexual misconduct policy. Third, additional items and instructions will be added to differentiate the presence of content of institutional online social media and personal social media as well as greater specificity around the differing functions of the included social media platforms. These changes will be vetted through the school projects partners and revised by the evaluation and project team in fall 2019.

The project team plans to administer the checklist an additional time with the six participating schools in early winter 2019. Following this, baseline and follow-up data will be compared, which will be used to inform and refine the tool to better meet the needs of campus partners. Once the tool is refined, the plan is to evaluate it for reliability, validity, and effectiveness in prompting change on campuses. The intention is to make the revised tool available for free download to any interested campus along with technical assistance, offered to the best of the project team’s ability. Ultimately, the process of developing and piloting this tool has led us to believe that this type of assessment, which can be completed relatively quickly and entirely free of cost, may be valuable to other areas of the campus digital environment beyond sexual violence. It became clear that if campus’ policies related to sexual assault are extremely difficult to comprehend without advanced education, it may be likely that other policies and procedures could be similarly complex and hard to access. In the digital age, a large component of accessibility involves making needed information available in the location where users of this content are most likely to search for it. As such, campuses may greatly improve the accessibility, appropriateness, and relevance of their online content and presence through simple evaluations such as the procedure presented here.