Testing the Testers: Are Young men who Have Sex with Men Receiving Adequate HIV Testing & Counseling Services?

Abstract

Background

The United States Centers for Disease Control and Prevention (CDC) promote HIV testing every six months among young men who have sex with men (YMSM) to facilitate entry into the HIV prevention and care continuum. Willingness to be tested may be influenced by testing services’ quality. Using a novel mystery shopper methodology, we assessed YMSM’s testing experiences in three cities and recommend service delivery improvements.

Methods

We assessed YMSM’s experiences at HIV testing sites in Philadelphia (n=30), Atlanta (n=17), and Houston (n=19). YMSM (18–24) were trained as mystery shoppers and each site was visited twice. After each visit, shoppers completed a quality assurance (QA) survey to evaluate their experience. Data were pooled across sites, normed as percentages, and compared across cities.

Results

Across cites, visits averaged 30 minutes (sd=25.5) and were perceived as welcoming and friendly (70.9%). YMSM perceived most sites respected their privacy and confidentiality (84.3%). YMSM noted deficiencies in providers’ competencies with sexual minorities (63.4%) and comfort during the visit (65.7%). Sites underperformed on LGBT visibility (49.6%) and medical forms inclusivity (57.95%). Sites on average did not discuss YMSM’s relationship context (49.8%) nor provide risk reduction counseling (56.8%) or safer sex education (24.3%). Sites delivered PrEP information and counseling inconsistently (58.8%).

Conclusions

Testing sites’ variable performance underscores the importance of improving HIV testing services for YMSM. Strategies are recommended for testing sites to promote cultural sensitivity: funding staff trainings, creating systems to assess adherence to testing guidelines and best practices, and implementing new service delivery models.

Introduction

Improving rates of HIV testing among young men who have sex with men (YMSM) is a crucial public health strategy for stemming the increasing HIV epidemic in the United States and achieving the Ending the HIV Epidemic in the United States by 2030 initiative¹. Although both research and intervention efforts have been dedicated to strengthening HIV testing uptake among YMSM^2–6, fewer resources have been expended to ensure that counseling, testing, and referral (CTR) services are developmentally and culturally tailored and responsive to the needs of YMSM^7–9. This is particularly problematic as prior research^10,11 has noted that clients may feel more motivated to engage in repeat HIV testing or to adopt other prevention efforts (e.g. pre-exposure prophylaxis [PrEP]) when agencies offer high-quality services and circumvent structural barriers (e.g., medical mistrust; stigma)^12–14. At present, however, there are no systematic assessments to evaluate the quality of agencies’ CTR services, nor an understanding of the how YMSM clients perceive and react to the quality of testing services.

Quality assurance (QA) indicators offer agencies opportunities to systematically evaluate their service performance, examine whether staff are implementing protocols with fidelity, and identify actionable improvement strategies and/or disseminate models of “best practice” to other agencies¹⁵. Mystery shopping^11,16 is an action-oriented strategy whereby organizations can understand their performance across a systematically-obtained set of service indicators, as rated by the client base they intend to serve. Bauermeister and colleagues¹¹ employed a youth-driven mystery shopper assessment of 46 HIV/STI testing clinics in the Detroit metropolitan area. YMSM were trained as mystery shoppers and in the use of a psychometrically-sound quality assurance instrument examining clinics’ environmental characteristics, compliance with federal and state CTR protocols, and performance during testing and counseling. Quality assurance data showed variability in the depth and quality of CTR services. For example, while mystery shoppers reported overall satisfaction with the environmental characteristics of the clinics, they noted that test counselors infrequently and inconsistently ascertained YMSM’s motivations for testing, discussed risk reduction strategies, and/or helped set actionable prevention goals. These findings underscore the need for quality assurance strategies to identify opportunities to strengthen the delivery of culturally competent HIV/STI testing services for YMSM.

As part of the NIH Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN), we deployed mystery shopping in three cities with high HIV incidence (Philadelphia, Atlanta, and Houston) in order to characterize the quality of HIV testing services available to YMSM living in these cities. Our study had three objectives. First, we examined the proportion of HIV testing locations that offered accessible HIV prevention services to YMSM, as characterized by the sites’ availability of free, rapid HIV testing and walk-in appointments. We then recruited and trained YMSM to serve as mystery shoppers and visit identified HIV testing locations in these three cities. Finally, we examined sites’ quality assurance domains and tested whether differences emerged across cities.

Methods

Identification of HIV testing locations in each city

Using AIDSVu.org and Google, we created a list of sites in each city that offered free, rapid HIV testing. We recorded specified clinic information for any testing sites that had websites, then we called each site using a standardized phone script to collect information about hours, location, and availability and characteristics (i.e. cost) of HIV and STI testing services. Discrepancies were frequent between online and staff-provided information. We excluded health systems (e.g., hospitals) from our lists as they do not allow rapid, walk-in HIV testing. When the list of verified sites was complete, we mailed each site a standardized letter with information about the study detailing the proposed mystery shopping and offered an opportunity to decline participation in the study. In Atlanta, one testing site contacted study staff to request their site be excluded from mystery shopping. No sites in Philadelphia or Houston declined participation.

Mystery Shopper Recruitment

Recruitment Strategy

Mystery shopper recruitment was managed by the local ATN site in each city. Each ATN site recruited participants using various methods, including 1) contacting former research participants who had consented to be contacted for future studies; 2) posting study information on the ATN site’s Facebook page; 3) reaching out to community organizations that serve YMSM; 4) partnering with LGBTQ groups at local schools and universities; 5) attending local events for YMSM or LGBTQ youth; and 6) word-of-mouth.

Screening

Interested individuals completed an online screener survey to determine eligibility. Inclusion criteria were: 1) assigned male at birth; 2) currently identify as male; 3) aged 18–24 years (inclusive) at time of screening; 4) self-report as HIV-negative; 5) speak and read English; 6) report same-sex attraction; 7) reside in Philadelphia, Atlanta, or Houston; and 8) able to attend testing sites.

Enrollment & Consent of Mystery Shoppers

ATN site staff contacted eligible applicants to arrange an in-person meeting to provide additional study details and complete the informed consent. At the meeting, ATN site staff reviewed participation details and answered any questions. Staff then consented the participant and asked about their availability for the half-day mystery shopper training. Demographic characteristics for the mystery shoppers across the three cities are detailed in Table 1. Study procedures were reviewed by the University of North Carolina’s Institutional Review Board, as the IRB of Record. Reliance agreements across the partnering institutions were secured.

Table 1.

HIV Testing Sites’ and Mystery Shopper Participants’ Characteristics by ATN City

	Philadelphia	Atlanta	Houston
Mystery Shoppers	N=9	N=6	N=9
Age Range	19–24	21–24	19–24
Race/Ethnicity
African American	8 (88.9%)	4 (66.7%)	0
Hispanic/Latinx	0	0	8 (88.9%)
White	0	0	1 (10.1%)
Asian American/Pacific Islander	1 (10.1%)	2 (33.3%)	0
Site Characteristics
Sites identified	N=53	N=50	N=46
Free HIV Testing	53 (100%)	50 (100%)	46 (100%)
Walk-in appointment	38 (71.7%)	19 (38.0%)	19 (41.3%)
Rapid HIV test	30 (56.6%)	17 (34.0%)	19 (41.3%)
Sites shopped	38 (71.7%)	19 (38.0%)	19 (41.3%)
Sites deemed accessible* to youth	30 (56.6%)	17 (34.0%)	19 (41.3%)

Notes. Site accessibility was defined as a site that offered free, walk-in HIV testing using a rapid test at the time of a mystery shopping visit.

Mystery Shopper Training

Before mystery shopping began, study staff led a half-day in-person training at the ATN site. The training included basic information about HIV, protocol for an HIV test, PrEP information, any state- or city-specific HIV testing rules, and an example of what a high-quality testing experience would include. Additionally, mystery shoppers were shown videos of simulated testing visits, followed by a discussion of what was done well and what could be improved in each. Study staff also performed role-plays of different testing scenarios and patient-provider interactions to elicit additional conversation and for mystery shoppers to practice using the site assessment survey. Mystery shoppers were instructed to be honest about their sexual behaviors during their visits and to avoid creating false personas. This guideline was informed by prior research suggesting that providers tend to alter their dynamics with patients during standardized patient assessment^11,16,17. By providing an honest narrative, shoppers were able to avoid arousing suspicion or creating confusion that might lead to embarrassment or incongruous stories. The training was designed to enhance shoppers’ self-efficacy for refusing medical procedures and/or asserting their rights to providers. The training concluded with a comprehensive explanation of mystery shopping visit procedures from scheduling to completion of the site assessment survey. Participants were paid $100 for the 4-hour training.

Site Visits

Pre-visit procedures

Before each visit, ATN site staff determined which site the mystery shopper would visit (site visits were randomized by day) and gave the shopper a study iPhone. Mystery shoppers used the phone to request an Uber ride to the testing site using the study’s Uber for Business account. Study phones protected mystery shoppers’ privacy and Uber facilitated easy transport to and from the testing sites. Mystery shoppers were instructed to use the study phone to contact ATN site staff if they encountered any issues over the course of their testing site visit. Some shoppers scheduled a second site visit at the conclusion of their first site visit. Following the shopper’s last testing site visit of the day, he would return to the ATN site using Uber.

Visit procedures

Each testing site was shopped twice at varying times (e.g. morning versus afternoon or evening) and days (weekdays vs. weekends when applicable) by two different mystery shoppers. Shoppers arrived at their assigned testing site and asked for a free rapid HIV test. Shoppers completed the previously validated site assessment survey on the study phone immediately after the visit or on a computer upon their return to the ATN site. The assessment survey collected information on the duration of the visit, visibility of LGBTQ symbols and printed materials (e.g., rainbow flag; magazines; flyers), the clinic environment, and how well privacy and confidentiality were maintained. Shoppers also evaluated the providers’ discussions regarding relationship context, testing and counseling assessment, safer sex recommendations, and PrEP-related discussions (see Table 2). At the end of each assessment survey, shoppers were given the opportunity to provide an overall qualitative impression of their experience at the site. These open text fields enabled the shopper to describe how they felt about the site and the provider; anything notable that had occurred over the course of the visit, be it positive or negative; and any other information deemed pertinent to the experience that was not already captured by the quantitative assessment. We offer illustrative quotes from shoppers across QA domains in Table 3.

Table 2.

Quality Assessment Domains across ATN Cities

		N	Mean	SD	Minimum	Maximum	F statistic	p
Visit Time (minutes)	1 PHL	30	30.05	32.20	7.5	180	1.21	.31
	2 ATL	17	35.50	20.67	8.5	100
	3 HOU	19	25.79	15.99	5.0	60
	Total	66	30.23	25.53	5.0	180
LGBT Visibility (α=.87; 2 items)	1 PHL	30	43.33	42.51	0	100	3.44a,b	.04
	2 ATL	17	70.59	36.70	0	100
	3 HOU	19	40.79	41.00	0	100
	Total	66	49.62	41.95	0	100
Medical Form Inclusivity (α=.91; 2 items)	1 PHL	30	53.33	46.76	0	100	1.44	.25
	2 ATL	17	73.53	42.82	0	100
	3 HOU	19	51.32	46.00	0	100
	Total	66	57.95	45.82	0	100
Clinic Environment (α=.87; 5 items)	1 PHL	30	71.60	17.29	30	100	3.12 b	.05
	2 ATL	17	79.65	14.46	53	100
	3 HOU	19	62.05	29.15	7	100
	Total	66	70.92	21.52	7	100
Privacy/Confidentiality (α=.60; 4 items)	1 PHL	30	86.03	13.46	50	100	.50	.61
	2 ATL	17	87.00	12.88	50	100
	3 HOU	19	79.05	31.48	0	100
	Total	66	84.27	20.18	0	100
Relationship Context (α=.75; 4 items)	1 PHL	30	51.40	29.12	0	100	.19	.82
	2 ATL	17	45.82	30.34	0	88
	3 HOU	19	50.84	38.58	0	100
	Total	66	49.80	31.98	0	100
Risk Reduction Counseling (α=.78; 5 items)	1 PHL	30	62.07	32.76	0	100	.95	.40
	2 ATL	17	48.12	33.36	0	100
	3 HOU	19	56.21	33.97	0	97
	Total	66	56.79	33.24	0	100
Safer Sex Education (α=.82; 3 items)	1 PHL	30	29.50	32.05	0	100	1.10	.34
	2 ATL	17	15.76	29.19	0	100
	3 HOU	19	23.74	29.57	0	100
	Total	66	24.30	30.69	0	100
PrEP Info & Dialogue (α=.86; 3 items)	1 PHL	30	67.73	35.78	0	100	2.05	.15
	2 ATL	17	44.06	39.86	0	100
	3 HOU	19	57.89	40.52	0	100
	Total	66	58.80	38.87	0	100
Perceived Provider Competencies (α=.86; 2 items)	1 PHL	30	66.67	30.18	0	100	.29	.75
	2 ATL	17	60.29	32.96	0	100
	3 HOU	19	61.05	33.86	0	100
	Total	66	63.41	31.63	0	100
Patient-Provider Interactions (α=.77; 5 items)	1 PHL	30	68.47	27.13	10	97	.90	.41
	2 ATL	17	69.47	32.17	17	100
	3 HOU	19	58.05	29.27	3	100
	Total	66	65.73	29.07	3	100

Notes. PHL: Philadelphia; ATL: Atlanta; HOU: Houston. All scores are aggregated by site and normed as percentages.

^arefers to a significant pairwise mean comparison between Philadelphia and Atlanta sites using a Tukey post-hoc correction test.

^brefers to a significant pairwise mean comparison between Houston and Atlanta sites using a Tukey post-hoc correction test. Reliability coefficients are estimated using Cronbach’s alpha for continuous variables or SK-20 for dichotomous variables.

Table 3.

Exemplary quotes from Mystery Shopper debriefing notes across domains

	Positive Feedback	Negative Feedback
Visit Time (minutes)	“The location was very close, and the counselor meeting was super quick. While one counselor was giving me the test, I was answering questions that the second person was asking me. This was a super quick rapid test with no waiting time. I was in and out within 15–20 minutes which is really nice and convenient. I would recommend this clinic for people nearby, friendly atmosphere, and very accessible. I feel that there wasn’t really much for improvement. Maybe just make lgbtq signs /cards flower more obvious and apparent.”	“The office was way too packed. It was full before they even started testing and the line went out of the door, which irritated the hall monitor. She asked us all to wait downstairs until we were called up. I would probably not recommend this site to get tested.”
Clinic Environment	“The staff were super nice, I felt very comfortable in the waiting room. I asked the receptionist about free hiv testing and she offered me testing for other stds/stis such as chlamydia etc”	“I only met two staff members but my interactions with both were not friendly. They seemed to be there only to do a job. The tester seemed almost impatient at times during the visit. I got no smile he didn’t ask me how my day was. This visit seemed to be a cookie cutter routine visit for the tester. The tester did give information about prep when I asked what prep was. But he seemed to be shocked when I reported I had not heard of prep. He offered me information on how to get on prep and gave me people to contact to and the time to contact them.”
Privacy/Confidentiality	“This clinic is very nice from the outside and inside. I also liked the efficiency of the workers and the helpfulness of the workers. There was sufficient privacy because we were taken into the consulting room for extra privacy and I liked how every thing was handled professionally. There was no judgement and were quick with everything which is why I would advertise this as one of Georgia’s best clinics that I have gone to.”	“They need to update the confidential part of it because I got tested in a room with people” “Lady at the desk was rude and said those services weren’t available to the public she was very loud and said HIV testing is not free, no confidentiality”
Relationship Context	“They were fast, affective, and nice. The staff made me feel comfortable and made sure I had enough resources to stay safe with my partner”	“The staff was very friendly and the building was well lit and had signs that guided me. My tester was very friendly and made me feel at ease during the experience. He asked me how my day was going and made small talk. We did not discuss why so was getting tested today or how many partners I am intimate with.”
Risk Reduction Counseling	“This was an awesome sight. The staff was very friendly. My counselor was spectacular. He really made me feel comfortable and made the experience one that wasn’t scary at all. My counselor offered me prep and even helped me set a date for my appointment. They also had a series of skits playing on the television that seemed to be aimed at helping people make smarter and safer decisions about their sex lives. I would definitely recommend this sight again.”	“Did not talk about sex or my reason for testing.”
Safer Sex Education	“The lady was extremely nice and made me feel comfortable and explained a lot of safer choices for me and providing me information about prep and asking me questions to further assist me!”	“I was offered no risk reduction options or condoms and or lube.”
PrEP Information & Dialogue	“Place had referrals for different programs such as housing assistance job prep resume building ....tokens and gift card incentives are offered for those who get tested and they also have prep referrals”	“It was very shaky. When I walked in there was two people at the front desk, one was sleeping and the other was on a phone call. They looked as if they had a really long day so I didnť blame them. The office space was very......dry like their felt sad. They tester kept pushing me on to PrEP and was a little disorganized. They looked very disorganized and the vibe was that they donť really have that many visitors. The visit was ok it just was shaky”
Perceived Provider Competencies	“Amazing friendly staff. Visit was quick and effective. The staff was knowledgeable about different LGBTQ issues. An amazing visit the whole time”	“I feel like my experience here was the typical experience of going somewhere where they see a lot of gay people and even work with them but there's a lack of real understanding of a lot of what comes along with the lifestyle - or someone who is passionate about there job but you're not really sure if it goes past the pay check or if they are in it because they are passionate and feel a sense of sympathy for the ‘project’”
Patient-Provider Interactions	“He appeared to know about HIV pretty well and he was able to tell me a lot about it in a way it felt like anyone would understand. Everything was fast overall.” “I really liked it, [provider] was very nice and super helpful. I felt so comfortable with him and it felt like I could just tell him and ask anything knowing he wouldn’t judge me, he takes his job seriously and he’s just so nice and caring.”	“They asked me for my id and checked me in using that, they also took my is and made a copy of it. Also the staff was not as friendly the lady testing me was very rude and insisted that I wasn’t being truthful about my reason for being tested which I shouldn’t feel forced to disclose more than I was comfortable with. I had told her ample info as to why I was getting tested and she kept probing for more and more almost making me feel as though she felt I was lying about my sexual activity which was an uncomfortable and aggravating feeling.”

Upon return to the ATN sites after a testing visit, mystery shoppers returned the study phone and a staff member conducted a short debriefing meeting. The informal interview served to capture any details of the visit that may have been missed in the site assessment survey, to help ensure any untoward events were discussed, and to offer any resources that may assist the mystery shopper. If possible, staff scheduled additional site visits with the shopper at the conclusion of the debriefing session. Alternatively, staff and participants would communicate via text or email to schedule additional site visits. After this quick conversation, participants received a $50 incentive for their completed visit. Shoppers could complete a maximum of 10 visits for this study.

Once mystery shopping was completed in each city, we mailed each site their personalized results, which included a summary of their pooled results on each of the QA domains assessed and their scores relative to other sites visited in the same city. The report also included mystery shoppers’ reflections, as noted in their evaluation surveys, and edited debriefing notes.

Data Analytic Strategy

Using the pre-established domains, we computed site composite score using the pooled scores between the two shoppers. Pooled scores are presented to reduce potential selection bias and confounding based on whether the same or a different provider interacted with shoppers at either site visit, as well as to account for the variability across shoppers. To ease interpretability across domains, we standardized the pooled average scores into percentiles. We used one-way ANOVAs with a Welch test correction to avoid assuming that the variance across scores would be comparable across the three cities. When appropriate, Tukey pair-wise comparisons were used to estimate differences between sites.

Results

Philadelphia

We originally identified 53 sites within the jurisdiction of the Philadelphia Department of Public Health (see Table 1). During verification of sites, we excluded 15 sites because they did not provide walk-in appointments, did not offer rapid HIV testing at their location, or were part of a health system. Thus, our sampling frame for mystery shopping in Philadelphia was 38 sites. 30 of the 38 sites visited were deemed youth accessible (i.e. site offered free, walk-in HIV testing using a rapid test at the time of a mystery shopping visit) upon completion of the mystery shopping procedures.

Philadelphia site visits averaged 30 minutes (sd=32.2) and were perceived to be welcoming and friendly environments (71.6%). YMSM perceived most sites respected their privacy and confidentiality (86.0%). YMSM noted some deficiencies in providers’ competencies in working with sexual minority youth (66.7%) and comfort during the interaction (68.5%). HIV testing sites were rated poorly by YMSM on LGBT visibility (43.3%) and inclusivity on medical forms (53.3%). During the risk assessment, Philadelphia sites on average did not discuss YMSM’s relationship context (51.4%), nor provide risk reduction counseling (62.1%) or safer sex education (29.5%). Delivery of PrEP information and counseling was also inconsistent across sites (67.7%).

Atlanta

We originally identified 50 sites within a 25-mile radius from Atlanta City Hall (see Table 1). During verification of sites, we excluded 31 sites, as most of the HIV testing sites were embedded within health systems, required a fee to test for HIV, did not offer rapid HIV testing, and/or did not offer walk-in appointments. Thus, our sampling frame for mystery shopping in Atlanta was 19 sites. 17 of these 19 sites visited were deemed youth accessible (i.e. site offered free, walk-in HIV testing using a rapid test at the time of a mystery shopping visit) upon completion of the mystery shopping procedures.

Site visits in Atlanta averaged 35.5 minutes (sd=20.7). Atlanta sites scored highest in YMSM’s perceptions that sites were welcoming and friendly environments (79.7%), with their scores being significantly higher than Houston sites (see Table 2). YMSM perceived most sites respected their privacy and confidentiality (87.0%). YMSM noted some deficiencies in providers’ competencies in working with sexual minority youth (60.3%) and comfort during the interaction (69.5%). Compared to the other two cities (see Table 2), Atlanta fared better in HIV testing sites’ LGBT visibility (70.6%). Atlanta sites were also perceived to have more inclusive medical forms (73.5%). During risk assessments, however, Atlanta sites did not regularly discuss YMSM’s relationship context (45.8%), nor provide risk reduction counseling (48.1%) or safer sex education (15.8%). Delivery of PrEP information and counseling was also inconsistent across sites (44.1%).

Houston

We originally identified 46 sites within a 25-mile radius from Houston City Hall (see Table 1). During verification of sites, we excluded 27 sites because they required a fee to test for HIV, did not offer rapid HIV testing, were a part of a health system, and/or did not offer walk-in appointments. Thus, our sampling frame for mystery shopping in Houston was 19 sites; all of these sites were deemed accessible to youth after their mystery shopping assessment.

Houston site visits averaged 25 minutes (sd=16.0), were perceived to be somewhat welcoming and friendly (62.1%) and respectful of YMSM’s privacy and confidentiality (79.5%). YMSM noted some deficiencies in providers’ competencies in working with sexual minority youth (61.1%) and comfort during the interaction (58.1%). HIV testing sites were rated as underperforming on LGBT visibility (40.8%) and inclusivity on medical forms (51.3%). During risk assessments, sites on average did not discuss YMSM’s relationship context (50.8%), nor provide risk reduction counseling (56.2%) or safer sex education (23.7%). Delivery of PrEP information and counseling was also inconsistent across sites (57.9%).

Discussion

Delivery of quality culturally and developmentally appropriate services may help YMSM access and navigate complex healthcare systems and achieve ongoing engagement with HIV prevention or care services. As an HIV test is often an individual’s first experience with clinical HIV prevention services (e.g., PrEP), it is important that this event is a positive experience, as a negative experience may deter future testing or engagement in care. At present, however, limited research has explored the quality of HIV/STI testing services. In the absence of such data, ensuring that HIV testing services offer developmentally and culturally competent care for YMSM remains a challenge. Therefore, this study’s systematic collection of YMSM’s experiences when seeking HIV testing and prevention services in three large metropolitan areas is an important step toward improving these services for youth.

While national campaigns and local efforts have sought to compile and share online HIV testing site locators^18,19, this study found that actual service availability at testing sites differed significantly from the service information provided in online resource directories. Given that HIV testing is one of the four pillars of the Ending the HIV Epidemic in the United States by 2030 initiative¹, these service availability discrepancies are particularly concerning as current guidelines⁶ call for YMSM to test for HIV every six months. It remains a priority to increase funding allocations to support the availability of agencies that provide free, same-day rapid HIV testing in highly-impacted areas identified in the Ending the HIV Epidemic in the United States by 2030 initiative.

Beyond increasing the availability of free, rapid, walk-in HIV testing locations across the country, it is paramount that efforts also strengthen the current delivery of CTR for YMSM. Encouraging HIV/STI testing among YMSM within public health campaigns may be less effective if HIV testing sites’ services are not culturally competent to this population^8,20–22. Overall, YMSM participating in the mystery shopping assessment found large variability in clinics’ efforts to showcase LGBT-friendly materials and offer LGBT-inclusive language on medical forms. This variability differed by city, with YMSM in Atlanta reporting greater visibility of LGBT-inclusive materials than those in Philadelphia or Houston. Given that prior research has recommended the integration of LGBT-affirming materials as a strategy to encourage comfort for sexual and gender minority populations within medical settings^21–23, renewed efforts focused on creating spaces that are welcoming to LGBT clients may be warranted.

The breadth and quality of the CTR sessions were also found vary across agencies in the three cities. Although most YMSM felt that sites maintained their privacy and confidentiality during visits, we were surprised by providers’ limited assessments of YMSM’s relationship contexts and discussions of safer sex education and PrEP. Moreover, YMSM perceived that providers could improve the quality of their interactions with them by offering risk reduction counseling, avoiding judgments during sexual risk screenings, and having greater awareness of the needs and experiences of YMSM clients. Taken together, these findings highlight the need to strengthen agencies’ delivery of culturally competent care through systems-level interventions¹⁵. The Health Access Initiative (HAI), for example, is a systems-level intervention that includes cultural responsiveness training for medical, clerical, programmatic, and administrative staff, as well as technical assistance to improve policies and protocols related to LGBTQ+ youth’s health care needs^24,25. At 6-month follow-up, clinics who received HAI training reported changes in their infrastructure (e.g., changes to medical forms, access to gender-inclusive bathrooms) and improvements in staff knowledge, attitudes, and practices when working with young sexual and gender minority clients. If we are to achieve the Ending the HIV Epidemic in the United States by 2030 initiative goals, investment in system-level interventions and strategies will be necessary to optimize HIV prevention and care delivery for YMSM.

This study has several limitations worth noting. Although we sought to diversify the data obtained within each testing site by having two mystery shoppers visit each site on different days and times, site assessments are unlikely to reflect all providers at each testing site. Second, our study does not seek to be generalizable to all testing sites across the United States, as each city may have a unique set of characteristics that influence the availability and quality of testing sites. However, our QA evaluation tool appeared to work well across three cities and had strong psychometric properties across the different domains. Third, we excluded health systems (e.g., emergency rooms in hospitals) from our mystery shopping procedures, which may not reflect the full landscape of HIV testing locations in a city. Finally, the cross-sectional design of our study limits our ability to make causal inferences about the data reported here. Future research examining the validity of the tool in predicting YGBM’s likelihood to repeat test and/or seek treatment in certain locations is warranted.

Despite these limitations, the findings of this study suggest that the implementation of youth-driven quality assurance systems for HIV testing services is feasible and may offer opportunities to strengthen the delivery of culturally competent HIV testing practices. Achieving high rates of routine HIV testing among YMSM is likely a product of both demand and supply. Rather than assuming that low HIV testing rates among some YMSM may be due to limited motivation to engage in HIV prevention (demand), our findings underscore the importance of considering whether site characteristics and provider interactions are also influencing YMSM’s testing motivations and behaviors (supply). Although mystery shopping is a novel strategy in the HIV testing context, we recognize that government agencies and community groups may have challenges implementing this strategy. Nevertheless, our developed indicators may be used by HIV testing sites and/or funders to assess site performance. For example, agencies could integrate the QA evaluation tool in their exit satisfaction surveys and review them quarterly to prioritize areas of need with their staff¹⁶. Similarly, funders could ask agencies to collect these data and include them in their annual reports, use these data to incentivize sites with evidence of strong care, and/or provide technical assistance to sites that are underperforming. Future research examining the implementation of these systemic strategies and their effect on YMSM’s testing and engagement in care is warranted if we are to achieve the ambitious goals set by the Ending the HIV Epidemic in the United States by 2030 initiative.

Figure :

HIV Testing Sites and HIV Prevalence per 100,000 people in (a) Philadelphia, (b) Atlanta, and (c) Houston