Constructing a Web-Based Health Directory for Adolescent Men Who Have Sex with Men: Strategies for Development and Resource Verification

Abstract

Online health directories are increasingly used to locate health services and community resources, providing contact and service information that assists users in identifying resources that may meet their health and wellness needs. However, service locations require additional vetting when directories plan to refer vulnerable populations. As a tool included as part of a trial of a mobile life skills intervention for cisgender adolescent men who have sex with men (AMSM; ages 13–18), we constructed and verified resources for an online resource directory focused on linking young people to LGBTQ+ friendly and affirming local health and community social services resources. We collected information for 2,301 individual directory listings through database and internet searches. To ensure the listings aligned with the project’s focus of supporting young sexual minority men, we developed multiple data verification assessments to ensure community appropriateness resulting in verification of 1,833 resources suitable for inclusion in our locator tool at project launch (March 2018). We offer lessons learned and future directions for researchers and practitioners who may benefit from adapting our processes and strategies for building culturally-tailored resource directories for vulnerable populations.

INTRODUCTION

Adolescent men who have sex with men or who have same-sex attraction (from here on referred to as AMSM) often face psychosocial vulnerabilities that exacerbate disparities across several health domains including HIV and STI infection, psychological distress, and substance use, misuse, and substance use disorders (J. Bauermeister et al., 2018; Halkitis, Kapadia, & Ompad, 2015; Harper & Riplinger, 2013; B. Mustanski, 2015; B. Mustanski, Birkett, Kuhns, Latkin, & Muth, 2015; B. S. Mustanski, Newcomb, Du Bois, Garcia, & Grov, 2011; B. Mustanski, Van Wagenen, Birkett, Eyster, & Corliss, 2013). The ability to successfully buffer or compensate against these vulnerabilities requires youth to activate intrinsic assets (e.g., resilience, self-esteem) and to have access to health-promoting resources in their social and physical environments (Fergus & Zimmerman, 2005; Ungar, Liebenberg, Dudding, Armstrong, & van de Vijver, 2013). Although providing referrals and access to supportive agencies is a long-standing strategy to reinforce social support, education, and learning in the adolescent health literature (Ambresin, Bennett, Patton, Sanci, & Sawyer, 2013; Harper, Jamil, & Johnson, 2012; Mazur, Brindis, & Decker, 2018), access to health services that are welcoming and affirming of sexual minorities varies widely across the United States (US) (Bauermeister et al., 2019; Sirdenis et al., 2019).

Linking AMSM to agencies that provide culturally competent and affirming health and social services is critical for reducing sexual minority-related vulnerabilities (e.g., HIV, STI, psychosocial outcomes) and promoting overall health and well-being (Eisenberg, Erickson, et al., 2020; Eisenberg, Gower, Watson, Porta, & Saewyc, 2020; Watson et al., 2020). Access to culturally-appropriate resources is vital to ensure that AMSM do not experience sexuality-related barriers when accessing support and care services (Bauermeister, Eaton, et al., 2015; Bauermeister, Pingel, et al., 2015; Knight, Shoveller, Carson, & Contreras-Whitney, 2014; Philbin et al., 2014; Tanner et al., 2014; Wilkerson, Rybicki, Barber, & Smolenski, 2011). However, youth often report barriers to accessing key services – including self-stigmatizing attitudes around mental health, perceived stigma from providers, poor provider match, and lack of resource accessibility – particularly in rural areas (Gulliver et al., 2010). AMSM report avoiding communicating with providers about their sexual orientation and sexual health concerns due to fear of heterosexist bias, concern their sexual health information will be disclosed to parents and a perception that they will not receive equitable treatment in a health care setting due to their sexual orientation (Fisher, Fried, Macapagal, & Mustanski, 2018). Perceived and experienced stigma in health care settings impacts service utilization by sexual and gender minority (SGM) youth (Whitehead, Shaver, & Stephenson, 2016) and can lead to worse health outcomes compared to heterosexual peers (Office of Disease Prevention and Health Promotion, 2020). Given these concerns, many youth may hesitate to access services if they do not know whether their sexual identity will be affirmed. To mitigate potential stigmatizing experiences, youth may try to vet an agency through the internet, including utilizing online resources to evaluate the suitability and friendliness of organizations and providers prior to utilizing the resource (e.g., attending an in-person or telehealth visit).

Online resource directories are becoming more prominent in public health as a method for linking people to health information and care resources across health topics, including cancer care and support (American Cancer Society, 2020), vaccination (VaccineFinder, 2020), diabetes education (American Diabetes Association, 2020), behavioral health (Substance Abuse and Mental Health Services Administration, 2020), pre-exposure prophylaxis (PrEP; Siegler, Wirtz, Weber, & Sullivan, 2017), and sexual health prevention (Valdiserri & Sullivan, 2018). Resource locators are also often included as a feature in digital health interventions either as a stand-alone tool or combined with other components (Bauermeister et al., 2018; Biello et al., 2019; Muessig, Nekkanti, Bauermeister, Bull, & Hightow-Weidman, 2015; Muessig, Pike, LeGrand, & Hightow-Weidman, 2012; Siegler et al., 2017; Sullivan et al., 2019). The popularization of these directories, including those available to the public and those included within health research, has made searching and locating health resources more convenient. However, the process of developing and curating such directories is rarely described in full, making it difficult to standardize or replicate across studies, a notable exception being Siegler et al.’s description of the development of the PrEP Locator (Siegler et al., 2017). Similarly, details of how teams verify these resources and maintain directory information also is typically lacking. Therefore, we offer additional methodology for developing online directories, using a case study of the development of a resource locator for AMSM. Specifically, our paper documents the methods for developing a resource directory for AMSM living in four diverse regions across the U.S. while integrating discussion of the unique challenges, considerations, and opportunities for future programs seeking to build similar resources for sexual minority populations.

METHODS

Study Protocol Description

The online resource directory described here was developed for the iReach study (Bauermeister et al., 2018), a randomized controlled trial (RCT) testing the efficacy of a web-delivered life skills intervention targeting cognitive and behavioral HIV-related outcomes for AMSM, operationalized to mean individuals aged 13–18 years who were assigned male sex at birth, identify as male at the time of enrollment, and report attraction and/or sexual experiences with people of the same gender. iReach is a mobile-optimized web application (web app) intervention that aims to facilitate participants ability to lower their vulnerability to HIV infection via psychoeducational and motivational content addressing 14 key life areas relevant to HIV prevention (e.g., sexual health information, family, stress, education, etc.). The focus of this paper is on the iReach web app’s resource locator which provides information about nearby community resources which may assist in goal attainment. Participants can perform self-directed searches in the resource locator (Figure 1) and are also directed to the locator from psychoeducational content pages to identify resources related to specific life skill topics.

Figure 1:

Resource locator display within iReach

The goal of the iReach RCT involved recruiting AMSM from four high HIV prevalence regions across the US: (1) Chicago, IL to Detroit, MI; (2) Washington, DC to Atlanta, GA; (3) San Francisco, CA to San Diego, CA; and (4) Memphis, TN to New Orleans, LA. Regions were selected based on the 2016 HIV prevalence rate maps accessed via AIDSVu.org (Sullivan et al., 2020). Each region includes urban, suburban, and rural counties (Ingram & Franco, 2012).

iReach Locator Development Methods

The study team developed the iReach Locator to provide access to appropriate services as a pathway to reducing AMSM vulnerability to HIV by connecting them to organizations providing HIV prevention and care services as well as resources for navigating structural (e.g., access to housing, transportation, mental health care) and cultural (e.g., social support, LGBTQ+ affirming spaces) environments that can act as supports or barriers. We designed and constructed the resource locator via the five steps below (Figure 2). Accompanying tasks, processes, and key decisions for each step are provided:

1. Define the content, scope, and procedure for conducting the search.

   • Define the scope of information to be included in database resource listings.
   • Operationalize scope into search terms and accompanying inclusion and exclusion criteria.
   • Develop a search process, identifying data sources and any search limits (e.g. Review the first five pages of Google.com search results)..
   • Define a data verification process, including steps to verify the accuracy of collected information, as well as steps to verify information not available online (e.g. does an organization require parental consent to provide services to youth).
2. Design the directory user interface, including visual display and ease of navigation. Conduct the search and populate the directory database.

   • Implement the search, refining the process to address challenges and increase efficiency as possible.
3. Clean, standardize, and format results for display.

   • Establish criteria for cleaning, standardizing and formatting information for display within the user interface design.
   • Conduct data cleaning and standardization.
4. Conduct pre-study-launch verification.

   • Operationalize the verification plan into checklists and scripts to ensure the process is implemented systematically.
   • Conduct pre-launch verification procedures.
   • Finalize initial dataset for display.
5. Conduct ongoing verification post-study-launch.

   • Conduct ongoing verification processes to update and maintain resource directory listings over time.

Though presented sequentially, these phases overlapped and influenced each other. Given the over-arching study’s focus on meeting the needs of AMSM, decisions were regularly driven by a focus on developmental appropriateness of the services. We highlight key considerations of how these factors influenced the process in the Results section below.

Figure 2:

Key processes and major tasks in locator development.

Data Sources

The study team populated the iReach Locator database using publicly available data about organizations providing services selected for inclusion in the resource locator. Data were located and accessed using two complementary search processes: 1) conducting web searches using the Google search engine and, 2) supplemental searches using reputable national databases related to HIV prevention and care (e.g., AIDSVu; Sullivan et al., 2020; PrEP Locator; Siegler et al., 2017), mental health and substance use treatment (i.e., SAMHSA Treatment Locator), LGBTQ+ support (i.e., PFLAG chapter locator), nutrition assistance (i.e., Feeding America, foodpantries.org), and housing resources (i.e., Homeless Shelter Directory, transitionalhousing.org).

Timeline and Staffing

The study team began conceptualization and design of the iReach resource locator in August 2016. Database population was conducted from January 2017 through September 2017. Initial rounds of verification activities (mail verification of all organizations and phone verifications of a random 20% of organizations) and data cleaning occurred from September 2017 through February 2018. The iReach trial launched in March 2018. Post-launch, the remaining 80% of included organizations underwent phone verification at a rate of 20% every six months until all sites had been verified, starting in May 2018 and concluding in October 2020.

RESULTS

Defining the content, scope, and process for the resource search

The study team defined the scope and contents of the locator database based on the study’s aim of assisting AMSM to navigate structural barriers to HIV prevention and care. Organizations that provided the following services were identified as important for that aim: HIV/STI treatment and prevention services, including testing; mental health and substance use services; housing resources; food and nutrition services; resources focused on healthy relationships and interpersonal violence prevention; and LGBTQ+ community resources. The team then identified key types of information to provide to users about each included organization, prioritizing information that would support AMSM in autonomous decision-making about service utilization, including: type of organization, services offered, address and contact information, religious affiliation (if applicable), and any known parental consent requirements. To assist with data management, the team developed a coding system by which each organization was assigned a unique identifier. The coding system included a series of standardized numeric prefixes to be added to the unique identifier allowing organizations to be sorted and categorized by shared characteristics (e.g., organization type and geographic location). Table 1 provides the final database fields included.

Table 1:

Standardized database domains, variables, definitions, and variable examples for a database of service locations for adolescent MSM participants in a randomized study of the iReach app, 2018–2021.

Domain and variable name	Variable definition	Variable example
Contact information
Agency Name	Name of the resource or provider	University of Pennsylvania
Street Address	Street address resource is located at	418 Curie Blvd
City, State	City and state resource is located in	Philadelphia, PA
Zip Code	5-digit zip code	19104
County	County resource is located in	Philadelphia
Phone Number	Primary telephone number for resource, entries must have 10 digits. Additional phone numbers contain detail on their purpose (e.g. crisis line, etc.)	215-898-5000
Email	Email address of the resource or provider.	info@myireach.com
Website	Website link for the resource or provider	https://www.upenn.edu/
Region Code & Identifier	Unique identifier assigned to a specific resource. The first number identifies the region a resource belongs to. The second number indicates the resource category (HIV/STI, Mental Health & Substance Use, LGBTQ+ resources) the resource belongs to. Final three numbers are a unique code that identifies which resource it is.	1-1010
Connected to religious group	Whether the resource is affiliated with a religious organization or tradition
Requires parental consent for service	Whether the resource requires parental notification and consent in order to participate
Services offered
Offers HIV Testing	Whether the resource offers HIV testing and referral services
Offers STI Testing	Whether the resource offers STI testing and referral services, which can include testing for Chlamydia, gonorrhea, syphilis, herpes, trichomoniasis, and hepatitis B, C
HIV Treatment or Care Services Available	Whether the resource offers services or support related to HIV treatment and care; services offered can include linkage to care and case management services; individual counseling, and support groups
Offers PrEP/PEP	Whether the resource offers consultation and prescription services for PrEP and PEP
Mental Health Support	Whether the resource offers direct counseling or support services related to mental health and wellness; services can include individual counseling, support groups, crisis lines, medication management, and assessment and testing services
Substance Use Support	Whether the resource offers services related to substance use treatment and recovery; services can include inpatient and outpatient treatment centers, crisis call lines, social service centers, and support groups
LGBTQ+ Group	Indicates that organization is specifically focused on serving the LGBTQ+ community; resources include pride festivals, LGBTQ community centers, advocacy organizations, and support groups
Programing for LGBTQ+ Youth	Indicates that the resource offers programing specific to LGBTQ+ youth
Supportive for Families	Indicates that a resource offers services that are supportive of LGBTQ+ youth and their families. Must include explicit indication that organization serves LGBTQ+ people in order to qualify for this service category
School Resources	Indicates that this resource offers LGBTQ+ programing, services, or trainings focused on supporting youth or staff in schools. Must include explicit indication that organization serves LGBTQ+ people in order to qualify for this service category

The geographic scope of the database was defined by the RCT’s inclusion criteria which involved 109 counties across the four recruitment corridors. We operationalized this criterion into search terms by including all 109 county names as well as the 3361 zip codes (as of 2017) associated with these counties.

The process for conducting the search combined organizational and geographic inclusion criteria to result in a comprehensive list of organizations. The study team began by conducting a search using the Google search engine and then supplemented results with organizations identified through existing, reputable databases to fill in any gaps left by Google search algorithms. Google searches were conducted using a combination of resource-specific search terms identified by members of the project team.

In addition to defining the scope of the search, we developed a set of inclusion criteria designed to ensure appropriateness for inclusion. Criteria for inclusion were: (1) offered at least one of the services listed above, (2) located within one of the RCT counties, (3) indicated LGBTQ+ friendliness either online (e.g., listed services specifically for LGBTQ+ individuals on organizational websites or social media) or by positively answering a question about providing LGBTQ+-affirming services during the phone screening process, and (4) offered services to people age 13 to 18 years old., and (5) passed a mailed verification process.

We also developed a mail and phone verification strategy to confirm the results of the web search, assess an organization’s willingness to serve LGBTQ+ youth and provide an opportunity for organizations to opt out of participation. First, a postcard was mailed to each organization serving to identify incorrect or out-of-date addresses, to inform organizations that we intended to include them in a study resource list, as well as instructions for opting-out of the database. Second, the team developed a phone verification script to be used to contact organizations, confirm the data collected through the web search, and to inquire about the organization’s policies, attitudes, and willingness to serve LGBTQ+ youth. A random 20% of all organizations were selected to be verified by phone prior to launch, providing the opportunity to identify trends that would require correction prior to study launch (e.g., high rate of incorrect data, organizations that do not serve LGBTQ+ populations).

Conducting the search and populating the resource database.

The search process was conducted from January to September 2017 and included 17 team members. As the search was conducted, the team adapted the original protocol to meet constraints of the available data. For example, many national databases (e.g., SAMHSA Treatment locator, AIDSVu) could accommodate searches using zip code, but not county. These adaptations were reviewed and approved by project managers with decisions being guided by the principle of including as much relevant data as possible in the final search results. As organizations were identified for inclusion, staff manually recorded site information in a standardized data spreadsheet (variables shown in Table 1). Table 2 presents the total number of resources collected.

Table 2:

Total resources identified by region and service type

Total Resources Identified, All Sources	2301		Total Resources Listed in iReach at 3/1/2018 Launch	1833
Region 1	955		Region 1	798
Region 2	669		Region 2	533
Region 3	449		Region 3	328
Region 4	221		Region 4	170
National Hotlines	7		National Hotlines	4
By Service Type			By Service Type
HIV & STI resources	855		HIV & STI resources	791
Mental Health resources	1035		Mental Health resources	951
Substance use resources	446		Substance use resources	418
LGBTQ+ resources	320		LGBTQ+ resources	304

Cleaning and formatting resource data for display in online locator

After completing the search, we cleaned, standardized, and formatted the results into a single list with uniform appearance across listings. We identified and combined duplicate listings of organizations. Staff contacted the organization to clarify potential duplicate listing that were ambiguous, resulting in 247 removals of duplicate entries. Staff also identified entries with missing data and either completed the entry or deleted the record if complete information was not available. Finally, staff reviewed entries to ensure they met inclusion criteria; we removed organizations that did not meet all inclusion criteria (e.g., not located within an approved county) Resources that were not located within an approved county were marked for removal (n=3).

Next, we formatted the data for display within the iReach web app, standardizing variations in contact information (e.g., telephone number format, 5-digit zip code format) to allow app developers to create an automated database update feature. This allowed study staff to quickly update entries by uploading a new spreadsheet. At this point, we also identified that some service information logged by the staff contained excessive details that might be unnecessary for AMSM population (e.g., overly detailed listing of specific STI or mental health services), therefore, we simplified the data to provide practical information for potentially topic-naive audiences. For example, rather than explaining each type of service offered, such as “individual therapy” and “group therapy,” this information was simply listed as “mental health resources.”

Conducting pre-study-launch verification

Mail Verification and Results.

In October 2017, we mailed postcards to all organizations to confirm physical address, inform agencies about the iReach project, and provide an option for organizations to request to be removed from our locator.

Between October 2017 and March 2018, we received 135 (6% of 2,301 mailed) postcards returned for the following reasons: “Not Deliverable As Addressed” (n=61), “No Mail Receptacle” (n=16), “Attempted – Not Known” (n=16), “No Such Number” (n=12), “Insufficient Address” (n=8), “Forward time Expired” (n=6), “No Such Street” (n=4), “Vacant” (n=4), “Unable to Forward” (n=3), “Unclaimed” (n=3), “Refused” (n=1), and “Return to Sender” (n=1). We removed these organizations from the resource database. Three organizations contacted the study team to learn more about the project or to update their contact information. No organizations opted out through the postcard mailing verification.

Phone Verification Process and Results.

Due to the high volume of organizations identified for inclusion in the database, verifying all by phone prior to launch was impractical. Therefore, prior to study launch, the study team randomly sampled 20% of organizations for pre-launch phone verification to both verify contact information and services offered and to provide the opportunity to verify AMSM-friendliness of the organizations included in the database. As noted below, the remainder of listed agencies were verified throughout the study period using this random selection process.

We trained two research coordinators and four research assistants to conduct verification phone calls using a standardized phone script (Appendix A) using multiple contact attempts. The script guided staff to verify all relevant database fields contained in the iReach database (Table 1). Staff training involved practice scenarios to help them prepare for unique situations (e.g., contacting staff with limited time, or navigating a switchboard or phone tree call system). To assess the “LGBTQ+ Group” service category, a specific question about the organizational mission or purpose was asked to help better understand the primary population served. Staff made pre-launch verification phone calls between November 2017 – February 2018 for 443 randomly selected resources: region 1 (n=186), region 2 (n=126), region 3 (n=90), and region 4 (n=41). Phone verification resulted in 19% (83/443) of resources marked for removal (i.e., 31% of region 1, 40% of region 2, 22% of region 3, and 7% of region 4). Reasons for removal were: inability to reach the organization after multiple phone verification attempts, organizations providing services exclusive to another population (e.g. adults, pregnant women, etc.), uncertainty or negative responses to questions about LGBTQ+ friendliness, opt-out requests, providing services outside the scope of the iReach project, only offering services contingent on court or social service involvement, and other reasons not specified.

Harmonizing Verification and Standardization Processes.

Once the entry standardization and duplicate removal had been completed, staff incorporated results of the phone and postcard verifications into the final database. Information from phone verification was prioritized. Resources that were marked for removal through the phone or postcard verification process were identified and removed from the final database list. If a listing was merged in the final list, project staff reviewed the reason for removal provided in the verification tracking sheet and discussed removing or maintaining the merged listing with project managers until consensus was reached. Combination of verification results was conducted by a single staff member during February 2018. Figure 3 presents the results of the two-part verification process, listing and location standardization, and the final number of listings included in the iReach Locator.

Figure 3:

Resource verification results

Uploading Verified and Standardized Database.

The final dataset was uploaded to the iReach resource locator on March 1^st, 2018. Figure 1 shows the final locator display within the iReach WebApp. A total of 1833 unique resources appeared in the locator at the launch of the trial. Final listings (Figure 1) in the iReach locator contain several design elements aimed at assisting young people in utilizing the tool as well as connecting with resources. The app uses responsive design features to allow participants continuity in their experience of the app regardless of their device. Contact information (e.g. website and phone number) and app elements are mobile friendly and tappable. Location sharing with the app is available as a way for participants to locate easily the nearest resource to their current location. Lastly, listings are visualized in simplified format to provide information in a straightforward and easily understandable format for the user.

Ongoing verification post-study-launch

Post-launch the study team completed phone verification of the remaining 80% in four phases, completing a fourth of the remaining calls every 6 months between May 2018 and October 2020. During the first round (May 2018 – October 2018) of post-launch verification phone calls, staff contacted 380 randomly selected resources: region 1 (n=159), region 2 (n=113), region 3 (n=69), and region 4 (n=39). Phone verification resulted in 17% (64/380) of resources being marked for removal (i.e., 37% from region 1, 35% from region 2, 18% from region 3, and 7% from region 4).

During the second round (November 2018 – September 2019) of post-launch verification phone calls, staff contacted 369 randomly selected resources: region 1 (n=156), region 2 (n=108), region 3 (n=68), and region 4 (n=37). Phone verification resulted in 6% (23/369) of resources being marked for removal (i.e., 47% from region 1, 21% from region 2, 21% from region 3, and 8% from region 4).

The third round of the post-launch verification phone calls (October 2019 – January 2020) contained 382 randomly selected resources: region 1 (n=168), region 2 (n=115), region 3 (n=67), and region 4 (n=32). Phone verification resulted in 10% (39/382) of resources being marked for removal (i.e., 64% from region 1, 23% from region 2, 12% from region 3, and none from region 4).

Staff contacted 376 randomly selected resources: region 1 (n=169), region 2 (n=109), region 3 (n=65), and region 4 (n=34), during the final round of post-launch verification phone (January 2020 – October 2020). Phone verification resulted in 4% (16/376) of resources being marked for removal (i.e., 75% from region 1, 12% from region 2, 12% from region 3, and none from region 4).

Locator usability

Among the 475 randomized participants who had engaged with the intervention, 193 (38.7%) have accessed the resource locator feature of the intervention to date. Among those accessing the locator, participants have viewed on average four resources (M=3.96, SD=5.52), with a range of resources viewed spanning between 1 and 43. Over a third of participants (n=71; 36.8%) viewed national resources. Most participants also viewed resources within their geographic area (i.e., 94.4% in region 1, 89.3% in region 2, 91.2% in region 3, and 86.9% in region 4). A chi-square test of independence showed that there was no significant association between using the locator and participants’ region, X² (4, N = 475) = 4.06, p = .26.

DISCUSSION

The inclusion of online resource directories within e-health interventions has gained popularity over the past 10 years (Vollmer Dahlke, Kellstedt, & Weinberg, 2015), yet detailed information on the process of developing and curating these directories remain limited in the literature. As a result, researchers are left to start the creation and curation of resource guides de novo, limiting the ability to standardize or replicate across studies and potentially increasing costs and delays in the absence of sharing lessons learned with the field. Similarly, details of how teams verify these resources and ensure the directory information remains updated is also often missing from published protocols. Thus, the goal of this paper was to detail the development of an online resource directory, using an intervention trial as a case study. While we recognize that the style and format of a resource locator will vary based on each project’s goals and the characteristics of its prospective users (e.g., sociodemographic information, psychological and developmental needs, etc.), lessons learned from our intervention planning process may assist future teams to systematically identify and verify resources in their communities.

Careful planning at the beginning of resource directory development is important, not just to ensure that the finished tool provides quality information, but also to make most efficient use of project time and resources. During these conversations, we were able to evaluate the strengths and limitations of various search methods (e.g., manual searches via search engine vs. utilizing existing databases), the information required to produce useful resource profiles, and the steps and effort required to verify collected information. As we report above, significant person hours went into collecting and verifying data to be presented in the iReach resource locator. We encourage teams to devote time early in the planning process to identify the purpose of the resource locator and the essential components that it should contain, as well as what resource listings are already available so as to optimize efficiency and allocation of effort.

Identifying the unique needs of prospective users is also critical. In our study, we were keenly aware that AMSM’s willingness to access local resources and their comfort in disclosing their sexual identity is likely influenced by how safe and affirming the organization and it’s representatives are (Philbin et al., 2014; Sirdenis et al., 2019; Tanner et al., 2014; Wilkerson et al., 2011). Thus, user needs and specific context provides important considerations for directory developers to take into account when designing verification processes that assist in thoughtfully linking users with potentially helpful resources. To identify sexual minority-affirming services, we used a two-part verification process. First, postcards were used as a low-cost strategy to verify mailing addresses listed online as correct and operational. This step also provided organizations an opportunity to opt-out of being listed in the locator. Second, phone call verifications, while more time and resource intensive, provided an opportunity to understand organizational and programmatic nuances. Through phone calls with organizations, study staff gained important insights in understanding possible experiences that young people might face when contacting or accessing services at an organization (e.g., friendliness of staff answering the phone, responses to direct questions related to LGBTQ+ competency). Future projects may consider using verification processes that not only confirm accuracy of information collected via the online search process, but also provide additional qualitative and contextual information about the resource being evaluated for inclusion in the directory.

Community resources and organizations are dynamic. Their operations and services are subject to change depending on a variety of factors including funding availability, shifts in organizational mission and purpose, and emergent community needs. Therefore, we implemented a prospective verification strategy where we selected a random subset of agencies to verify every six months over the phone. We found between 6% and 17% of resources reached each period noted changes to their services. These findings underscore the need to plan for ongoing updating and verification of resources listed in an online locator. Undoubtedly, the manualized identification of resources can be taxing in terms of financial and staffing resources, but we found this was essential to maintain the quality of resources offered to this vulnerable population. Therefore, we recommend that teams plan their verification plans based on their capacity to maintain these efforts based on the available staff, the total number of resources included in their directory, the expectation that agencies may change their focus over time, particularly if these resources are for a targeted population such as AMSM.

Future locators and verification processes should consider ways to provide ongoing and responsive updates to listings that more closely reflect the status of the organization. For example, with the advent of big data computational technologies and use of machine learning technologies, the financial and staffing resources to create and maintain the resource directories may be lessened. Machine learning algorithms can be used to scout the internet for emergent resources in a community, to update changes to agencies’ information (e.g., contact information, hours of operation) on the directory, and to create language classifiers that grade users’ online reviews of services to indicate quality and inclusivity. This approach may be particularly appealing to adolescent populations who rely heavily on the opinion of their peers. Future research examining the suitability and effectiveness of machine learning algorithms to create and maintain a resource locator is warranted.

To make the verification process viable, we asked a small set of questions during phone call verifications. Although our decision to keep our verification calls brief helped assist the research team in building a national resource database, we recognize that a larger assessment might strengthen our ability to assess an organization’s organizational attitudes and willingness to serve LGBTQ+ people with greater precision. As a result, the level of verification conducted during our study possibly overestimates a resources competence level when working with this population. More nuanced and refined verification processes, such as mystery shopping verification and/or standardized and confidential reviews by users who access a given service, may provide a more complete perspective of the agencies’ cultural competency. Future research examining the differential cost-utility and value added that is provided by these strategies is warranted and may inform future programmatic efforts.

Formalizing the locator creation and verification processes early in our study development provided a roadmap for our trial. As web-based directories become more popular as well as more complex, developers will need to give thought to the ways that information is collected, verified, and presented to users. For iReach, our decisions surrounding data collection and verification processes were focused on identifying relevant resources and reviewing the listings for suitability in our national geolocated resource directory. Tailoring our processes specifically to focus on the health and safety needs of AMSM provides a starting point within the literature to begin exploring new and innovative ways of reviewing information and resources included in directories. The methods discussed for data collection and tracking may also provide examples of effective strategies that directory developers can use to organize and track complex sets of data across multiple verification processes. Depending on the scale and scope of the directory being created, verification processes can be adapted, modified, or built upon to best match the needs of the locator’s intended audience. We hope that our experience serves as lessons learned for future studies seeking to create resource directories.

Appendix A: Final resource phone verification script

Phone Script for Resource Verification

This script is to help guide you through the resource verification process for the U01 iReach Project. Information from online searches needs to be verified via phone or email contact with each organization to ensure that we are providing participants with accurate information about resource they may need.

Verification attempts should first be made by phone calls. Phone calls should be made using office phones, not personal phones. When calling, use the phone script can help guide you through the verification process. The following highlighting guide should be used throughout the verification process.

Green = Verification complete and accurate

Purple = updated contact information (highlight only parts that have been updated)

Yellow = Started but not finished (highlight only parts that have been verified)

Red = Flagged for removal

Orange = Unable to reach, try again at a later time

Blue = to discuss with James or Ryan

No color = not yet contacted

If you are unable to reach an organization by phone, the agency should be flagged in the database by highlighting it in orange. A staff member will attempt to contact the agency.

If there are any discrepancies between the information pulled from the internet and what the agency provides over the phone, change the information in the resource list to match the information provided over the phone. If you have concerns or questions about including an organization or agency, highlight the resource in blue and discuss with a staff member to determine next steps. After the agency has been verified, an “x” should be marked in the “verified” column.

Phone Script (Agency Answered)

ASK, THE QUESTIONS BELOW. FOR EACH:

IF NO: –Request correct information and take notes—

IF YES: –Continue with next question—

“We have the name of your organization listed as {insert name of this organization here}. Is this correct?”

“Is your office address ( say Address) and is it located in (say county)?”

“Is the primary phone number ( say phone number) ?”

“If someone wants to email your organization, should they use ( say email address) ?”

“Is your website ( say website) ?”

AFTER ALL CONTACT INFO VERIFIED

“Great. And I have just a few questions about your organization”

“Does your organization require parental consent in order for minors to participate in your programs?” – Note response –

“Does your organization have a religious affiliation?” – Note response –

AFTER PARENTAL CONSENT AND RELIGIOUS AFFILIATION INFO IS VERIFIED:
“Now I have a few questions about the programs or services your organization offers:”
“Does your organization offer in-person groups or programs for youth?
IF YES: Do you offer any of the following:”
Sexual health services (If no, skip. If yes, ask the following):
HIV testing and counseling	Answer: Yes / No
STI testing and counseling	Answer: Yes / No
HIV treatment & care services	Answer: Yes / No
PrEP or PEP services	Answer: Yes / No
Mental Health or Substance Use Services (If no, skip. If yes, ask the following):
Mental Health Counseling	Answer: Yes / No
Support Groups	Answer: Yes / No
Substance use support	Answer: Yes / No
Programing on LGBTQ topics (If no, skip. If yes, ask the following):
Programing specifically for LGBTQ+ youth	Answer: Yes / No
Supportive programs for family	Answer: Yes / No
Resources that support groups or programs in schools	Answer: Yes / No
IF NO: -- Note that there are no in-person programs. Continue to next question.
“Does your organization offer any additional services or programs that I haven’t mentioned?”
IF YES: Note if it falls into one of the following categories:
Food pantry services	Answer: Yes / No
Shelter and housing support	Answer: Yes / No
Resources for intimate partner violence	Answer: Yes / No
Other services not listed –Take notes—
“Does your organization’s mission or purpose specifically focus on serving LGBTQ+ people?”
IF YES: Mark 1 in the LGBTQ+ Group column
IF NO: Mark 0 in the LGBTQ+ Group column and ask
“Is your organization affirming of LGBTQ+ people?
IF YES: Continue
IF NO: Flag for removal and pleasantly end the phone call
“Is there any other information about your programs that we should list in our directory?”
IF YES: –Take notes—
IF NO: –Continue—
“Thanks for taking the time to verify that information with me. If your organization has any questions in the future, you can contact us by emailing pstar@nursing.upenn.edu or call us at 844-849-7473. Your organization also should have received a postcard with additional information and information on how to opt-out of the resource list if your organization does not wish to be included. We can also remove your organization from the resource directory at this time if you do not wish to be listed.
IF they want to remain listed: Do nothing –Request correct information and take notes—
IF they want to be removed: –Highlight the resource in RED to flag for removal.
IF UP-TO-DATE: “Thank you. I appreciate your help. Goodbye” –End Call and mark as verified
IF NOT UP-TO-DATE: “Thank you. Goodbye.” (NOTE ANSWER AND FLAG FOR REMOVAL FROM LOCATOR).

Phone Script (Leaving a message if Agency Did Not Answer)

Hello, my name is ________ and I am calling from the University of Pennsylvania’s School of Nursing. We are currently putting together a list of organizations providing resources to adolescents on topics including health, education, and social support. As part of our search, we identified {Name of this organization} as an organization providing resources or services to adolescents. I am calling to make sure that the information we found is correct before including it in our list. We would appreciate if you were able to call us back at 844-849-7473 to verify your organization’s services. If you are unable to reach us, please leave a message with the best time to call you back. Thank you and have a wonderful day.

Email Follow-Up (Sent from PSTAR Admin Acct.)

Dear (Insert Agency Name),

We are the Program on Sexuality, Technology, and Action Research (PSTAR) at the University of Pennsylvania’s School of Nursing and are currently putting together a list of organizations providing resources to adolescents on topics including health, education, and social support.

We are trying to verify your organization’s information before we include it in our list. We reached out to you via phone on (Insert Date) but were unable to reach someone.

In order to ensure we are providing accurate information, we would appreciate if you were able to call us at 844-849-7473. The verification process should take no longer than 10 minutes of your time.

Thank you and we look forward to hearing from you soon.