The Use of Mystery Shopping for Quality Assurance Evaluations of HIV/STI Testing Sites Offering Services to Young Gay and Bisexual Men

Abstract

Young men who have sex with men (YMSM) are at increased risk for HIV and STI infection. While encouraging HIV and STI testing among YMSM remains a public health priority, we know little about the cultural competency of providers offering HIV/STI tests to YMSM in public clinics. As part of a larger intervention study, we employed a mystery shopper methodology to evaluate the LGBT cultural competency and quality of services offered in HIV and STI testing sites in Southeast Michigan (n = 43).We trained and deployed mystery shoppers (n = 5) to evaluate the HIV and STI testing sites by undergoing routine HIV/STI testing. Two shoppers visited each site, recording their experiences using a checklist that assessed 13 domains, including the clinic’s structural characteristics and interactions with testing providers. We used the site scores to examine the checklist’s psychometric properties and tested whether site evaluations differed between sites only offering HIV testing (n = 14) versus those offering comprehensive HIV/STI testing (n = 29). On average, site scores were positive across domains. In bivariate comparisons by type of testing site, HIV testing sites were more likely than comprehensive HIV/STI testing clinics to ascertain experiences of intimate partner violence, offer action steps to achieve safer sex goals, and provide safer sex education. The developed checklist may be used as a quality assurance indicator to measure HIV/STI testing sites’ performance when working with YMSM. Our findings also underscore the need to bolster providers’ provision of safer sex education and behavioral counseling within comprehensive HIV/STI testing sites.

Introduction

Consistent with national trends, men who have sex with men in Southeast Michigan are the only risk group for whom HIV and STI incidence has increased since 2000, with HIV incidence among young men who have sex with men (YMSM) nearly doubling [1]. YMSM accounted for 72 % of new HIV infections and were estimated to account for over 80 % of new syphilis diagnoses among young men ages 13–24 [2]. Given the existing HIV/STI vulnerability experienced by YMSM in Southeast Michigan, it is vital that public health programs and interventions encourage YMSM to follow CDC guidelines regarding HIV/STI repeat testing and early detection, linkage and retention in care. In order for these efforts to succeed, both HIV and STI testing sites must be sensitive to the needs of sexual minority young men. Delivery of culturally and developmentally adequate services may help these youth to access and navigate complex healthcare systems, and achieve continuous engagement with sexual health care services [3, 4].

HIV and STI testing initiatives for YMSM are crucial prevention strategies [5]. Over a quarter of HIV infected individuals in the United States do not know their status, and are estimated to account for a large proportion of new HIV transmissions [6]. Although both research and intervention efforts have been devoted to strengthen HIV/STI testing outreach among YMSM as a risk group [7–9], fewer resources have sought to examine and optimize the quality of these testing services [3, 4]. Furthermore, there is increasing recognition of the importance of integrating sexual identity-specific factors (e.g., sexual orientation identity) and barriers (e.g., stigma, prejudice) into the delivery of care. Compared to ongoing efforts focused on strengthening medical providers’ delivery of LGBT competent healthcare [10–13], there are limited resources focused on ensuring the quality and cultural competency of HIV/STI testing providers who work with young gay and bisexual men (YGBM). A specific discussion of YGBM’s HIV/STI testing needs from a cultural competency perspective is crucial, as it may provide insight into how to use identity-related processes (e.g., coming out, relationship status) to understand the client’s social context, and provide tailored risk reduction counseling beyond a non-specific risk reduction discussion based on their risk category (e.g., YMSM) [14]. Moreover, it remains unclear whether culturally competent care may vary as a result of the type of test performed. For example, service delivery may differ between HIV testing and comprehensive HIV/STI testing sites by setting (e.g., AIDS service organizations vs. local health department) and the tests’ invasiveness (e.g., rapid HIV test vs. phlebotomy and swab tests for STIs).

Assessments of cultural competency in public health practice are crucial to the optimal functioning of public health systems [15]; however, there are limited indicators to measure sexual health service providers’ ability to work with YGBM. In a web survey of LGBT youth ages 13–21 in the United States and Canada, Hoffman et al. [16] found that the two most salient issues regarding healthcare preference included having a provider who understands the issues of LGBT youth and who treats patients with respect. In a qualitative study with health care providers and LGBT patients, Wilkerson et al. [17] found that culturally competent clinical environments were evaluated based on environmental characteristics (e.g., LGBT-friendly décor), clinical components (e.g., language in forms), and interpersonal factors (e.g., patient-provider communication). Based on these prior studies, we sought to develop and test the psychometric properties in a set of indicators examining the aforementioned clinics’ characteristics, as well as YGBM’s experiences with providers during their testing and counseling (e.g., quality of testing and counseling session, safer sex discussions, perceived provider competency).

To examine HIV/STI providers’ performance on these indicators within HIV/STI testing sites across Southeast Michigan, we employed a mystery shopper approach to collect real-time data on each HIV/STI testing site visited. Commonly used in the hospitality and consumer industry, the mystery shopper approach [18–20] is increasingly used in public health evaluation. Similar to the announced standardized patient evaluation used in medical training, public health researchers have used mystery shoppers to evaluate healthcare delivery in community settings [21]. In 1992, Russell et al. [22] examined whether 65 primary care physicians offered a consultation on STI prevention to their mystery shoppers, described as sexually-active females new to the area. A third of physicians did not ask any risk questions, and over 80 % did not ascertain patients’ sexual risk behaviors nor offered safer sex education. The use of mystery shoppers has also been used to examine access to reproductive health technologies in pharmacies. For example, Sykes and O’Sullivan [23] trained young people as mystery shoppers to evaluate the provision of sexual and reproductive health care services in the United Kingdom. In their findings, they noted that standard practices regarding patient confidentiality were not followed by administrative staff (e.g., receptionists) working in these clinics. Similarly, French and Kaunitz [24] and Higgins et al. [25] used mystery shoppers to examine the availability of Plan B contraception, as well as staff’s knowledge about—and willingness to dispense—emergency contraception across pharmacies in Florida and Australia, respectively. Building on these prior studies, we implemented a mystery shopper evaluation of young gay and bisexual men across HIV and STI testing sites in Southeast Michigan.

The overall goal of this project was to evaluate the cultural competency of HIV/STI testing providers when working with YGBM in Southeast Michigan. Our study had three objectives. First, we assessed the performance of these indicators using a mystery shopper approach across 43 testing sites. Second, we used the data collected through the mystery shopper approach to evaluate the psychometric properties of our clinic and patient-provider indicators. Finally, we compared whether these indicators varied based on whether the HIV/STI testing location provided only HIV testing services or if the site provided comprehensive HIV/STI testing services.

Methods

Procedures

This project emerged as part of a community-based participatory research partnership [8] between University of Michigan researchers and two community boards: a Youth Advisory Board and a Provider Advisory Board. The University of Michigan team compiled an exhaustive list of information on HIV/STI testing clinics in Southeast Michigan, which included sites in the US Census’ Detroit-Ann Arbor-Flint Combined Statistical Area. Initially, we populated the list using CDC and Google databases of HIV/STI testing locations in Southeast Michigan. We then compared these data to clinics’ own websites when available. We cross-examined differences in clinic information such as STI testing hours, services provided, phone numbers, addresses, language used to describe clients and services, targeted client age, and cost of services. Next, each testing center was called via telephone to verify and reconcile the information.

Five male research assistants performed site evaluations at 43 locations throughout Southeast Michigan between April and July 2013. The research assistants’ were diverse in terms of their racial/ethnic identity (2 White/Caucasian; 1 Black/African American; 1 Indian; 1 White/Asian Pacific Islander) and matched the study’s target population in terms of sexual orientation, while the age range was slightly higher (22–31 years). Research assistants were hired to work an average of 10 h per week ($15 per hour) as part of this project. Research assistants conducted two separate evaluations at each testing location during varying times (e.g. morning versus afternoon or evening) and days (weekdays vs. weekends when applicable). A shared document was created for mystery shoppers to note when and where they would be getting tested. Study procedures were reviewed by the University of Michigan’s Institutional Review Board and ruled exempt.

Before site evaluation began, research assistants received self-efficacy training by a clinical psychologist in order to strengthen their ability to handle difficult situations in which they might find themselves at odds with providers. Specifically, training was provided on how to turn down medical procedures and/or assert their rights to providers, taking turns to role-play different scenarios and interactions that might occur. To protect their privacy, each research assistant was given a temporary, pre-paid ($100) mobile phone to use when scheduling appointments. We also gave each shopper a Zipcar account to ease their travel to the sites. On average, travel costs from/to a site were $50.

Mystery shoppers were instructed to be honest about their sexual behaviors during their visits and to avoid creating false personas. This decision was guided by prior research suggesting that providers tend to alter their dynamics with patients during standardized patient assessment [26]. Thus, by providing an honest narrative, shoppers were able to avoid arousing suspicion or creating confusion that might lead to embarrassment or incongruous stories. To ascertain sites’ ability to accommodate and address structural barriers often faced by youth of disadvantaged communities, we instructed the research assistants to indicate that they did not have any income, health insurance, or proof of identification during their visits. In so doing, the study team was able to ascertain whether these would be potential barriers to testing at a given location. For sites that had a sliding fee scale for testing, no income meant the lowest possible fee. They were also instructed to avoid any site that charged more than $60, as most YMSM would be unable to pay this amount for a comprehensive HIV/STI testing panel. We reimbursed the mystery shoppers for all charges linked to their HIV and STI testing experiences.

Research assistants completed a provider checklist immediately after each testing site visit. Building on qualitative domains proposed by Hoffman et al. [16] and Wilkerson et al. [17], we triangulated these findings with the lived experience of our youth advisory board and our community advisory board. From our discussions, we created a checklist that examined shoppers’ experience during their HIV/STI testing visit. This checklist collected information on how to schedule appointments, the number of days that it took to see a provider, the duration of the clinical visit, and proximity to a bus stop. The checklist also ascertained sites’ environmental characteristics, availability of youth and LGBT materials, clinic’s environment, and ability to maintain privacy and confidentiality (see Table 1). Finally, shoppers evaluated the providers’ discussions regarding relationship context, testing and counseling assessment, sex education recommendations, as well as negative reactions to partner-seeking behaviors (see Table 2). In addition, they provided their overall qualitative impression of the site—how they felt about the site and the provider; anything notable that had occurred in the course of the visit, be it positive or negative; and any other information deemed pertinent to the experience that was not already captured by the quantitative assessment. After their visit, the research assistants also completed a follow-up form that indicated how and when they had received their results from their visit. HIV-only sites provided rapid HIV tests, allowing the research assistants to complete the form on the same day; however, among comprehensive HIV/STI sites, we instructed the research assistants to complete the follow-up form once they were called by the agency. Research assistants waited a maximum of 14 days to allow for enough time to receive their STI results from each site. After each visit, the research assistants debriefed with one of the senior members of the research team.

Table 1.

Clinic characteristics observed across sites (N = 43)

	Total score (N = 43) M (SD)/N (%)	HIV-only (N = 14) M (SD)/N (%)	Comprehensive HIV/STI testing (N = 29) M (SD)/N (%)	t/χ2	p
Clinic characteristics
I can schedule an appointment via phone	39 (90.7 %)	13 (92.9 %)	26 (89.7 %)	0.12	0.74
I can schedule an appointment online	10 (23.3 %)	1 (7.1 %)	9 (31.0 %)	3.02	0.08
How much time does it take to wait for an available   appointment? (in days)	0.94 (1.69)	0.36 (.93)	1.22 (1.90)	−2.01	0.05
The provider is near public transportation	38 (88.4 %)	14 (100 %)	24 (82.8 %)	2.73	0.10
Session speed (min)	45.64 (20.58)	45.07 (22.34)	47.29 (20.57)	−0.31	0.76
Youth visibility (α = 0.95)
Clinic has posters aimed at youth	38 (88.4 %)	13 (92.9 %)	25 (88.4 %)	0.41	0.52
Clinic has printed materials (e.g., magazines/brochures)   aimed at youth	37 (86.0 %)	12 (85.7 %)	25 (86.2 %)	0.002	0.97
LGBT visibility (α = 0.84)
Clinic has posters aimed at LGBT people (e.g., equal sign,   pink triangle, rainbow flag)	16 (37.2 %)	7 (50.0 %)	9 (31.0 %)	1.45	0.22
Clinic has printed materials (e.g., magazines/brochures)   aimed at LGBT people	21 (48.8 %)	9 (64.3 %)	12 (41.4 %)	1.98	0.16
The clinic has LGBT welcoming symbols	16 (37.2 %)	6 (42.9 %)	10 (34.5 %)	0.28	0.59
Medical Forms (α = 0.59)
Clinic uses LGB-inclusive language on medical forms	40 (93.0 %)	12 (85.7 %)	28 (96.6 %)	1.71	0.19
Clinic uses transgender-inclusive language on medical   forms	36 (85.7 %)	11 (84.6 %)	25 (86.2 %)	0.02	0.89
Clinic environment (α = 0.76)a
The office staff were generally friendly	3.48 (0.52)	3.43 (0.58)	3.50 (0.50)	−0.42	0.68
The office staff were judgmental. (reverse-coded)	3.56 (0.58)	3.61 (0.53)	3.54 (0.61)	0.38	0.71
The office staff were not LGBTQ sensitive. (reverse-coded)	3.50 (0.51)	3.62 (0.51)	3.42 (0.51)	1.07	0.30
I felt uncomfortable in the waiting room. (reverse-coded)	3.12 (0.78)	3.36 (0.50)	3.00 (0.87)	1.43	0.16
Clinic used LGB-affirming language when speaking to   me.	3.22 (0.80)	3.46 (0.63)	3.07 (0.87)	1.49	0.15
Privacy and confidentiality (α = 0.29)
The office staff kept patient information confidential	37 (92.5 %)	14 (100.0 %)	23 (62.2 %)	1.75	0.19
My privacy was not violated while in the waiting room	39 (90.7 %)	13 (92.9 %)	26 (89.7 %)	0.12	0.74
The provider explained confidentiality (either verbally or   via a document)	38 (88.4 %)	13 (92.9 %)	25 (86.2 %)	0.41	0.52
The provider kept patient information confidential	41 (97.6 %)	14 (100.0 %)	27 (96.4 %)	0.51	0.47

^a4-point scale (1 = strongly disagree; 4 = strongly agree)

Table 2.

Patient-provider interaction indicators observed across sites (N = 43)

	Total score (N = 43) M (SD)/N (%)	HIV-only (N = 14) M (SD)/N (%)	Comprehensive HIV/STI testing (N = 29) M (SD)/N (%)	t/χ2	p
Relationship context (α = 0.69)
The provider asked me about my sexual orientation	19 (44.2 %)	5 (35.7 %)	14 (48.3 %)	0.60	0.44
The provider asked me about my relationship status	26 (60.5 %)	9 (64.3 %)	17 (58.6 %)	0.13	0.72
Provider asked if I experienced intimate partner violence	26 (60.5 %)	13 (92.9 %)	13 (44.8 %)	9.11	0.01
Counseling session (α = 0.76)a
The provider explored my motivation for testing	3.05 (0.91)	3.14 (0.81)	3.00 (0.95)	0.48	0.63
The provider offered to help me set goals	17 (39.5 %)	9 (64.3 %)	8 (27.6 %)	5.32	0.02
The provider offered to help me set action steps to meet new   safer sex goals	17 (39.5 %)	10 (71.4 %)	7 (24.1 %)	8.83	0.01
The provider offered me risk reduction options	30 (69.8 %)	12 (85.7 %)	18 (62.1 %)	2.50	0.11
The provider's recommendations were valuable	33 (82.5 %)	12 (92.3 %)	21 (77.8 %)	1.28	0.26
Notification of results (α = 0.45)
The provider explained how I could receive my results	41 (95.3 %)	14 (100.0 %)	27 (93.1 %)	1.01	0.31
The provider asked me how I would like to be contacted   regarding follow-up	27 (62.8 %)	6 (42.9 %)	21 (72.4 %)	3.53	0.06
Safer sex education (α = 0.88)
Provider made sure I knew how to use a condom	9 (20.9 %)	6 (42.9 %)	3 (10.3 %)	6.03	0.01
Provider helped me identify a condom that works for me	9 (20.9 %)	6 (42.9 %)	3 (10.3 %)	6.03	0.01
Provider helped me identify a lube that works for me	8 (18.6 %)	6 (42.9 %)	2 (6.9 %)	8.03	0.01
Perceived provider competency (α = 0.65)a
The provider/test counselor appeared knowledgeable about HIV/   STIs	3.40 (0.61)	3.43 (0.58)	3.38 (0.64)	0.24	0.81
The provider appeared knowledgeable about LGBTQ health   issues	3.09 (0.83)	3.36 (0.82)	2.90 (0.80)	1.62	0.11
Negative provider interactions (α = 0.89)a
The provider made me feel comfortable. (reverse coded)	1.80 (0.85)	1.93 (0.85)	1.74 (0.86)	0.67	0.51
I felt pressured by provider to adopt specific risk reduction   options	1.57 (0.84)	1.71 (0.95)	1.50 (0.78)	0.79	0.44
Provider was judgmental about the kind of sex I have (e.g., anal/   receptive/penetrative, etc.)	1.74 (0.91)	1.93 (1.12)	1.64 (0.77)	0.95	0.35
Provider was judgmental about how many partners I have had	1.60 (0.77)	1.82 (0.93)	1.54 (0.67)	1.05	0.30
Provider was judgmental about how I met my partners	1.81 (1.10)	2.00 (1.26)	1.64 (0.95)	0.82	0.42

^a4-point scale (1 = strongly disagree; 4 = strongly agree)

Data Analytic Strategy

Analyses performed included descriptive statistics for average site scores. Average scores are presented to reduce potential selection bias and confounding based on which provider was assessed within an agency at either visit, as well as to account for the variability across shoppers. For ordinal variables, we computed a site composite score for each indicator using the average score between the two shoppers. For dichotomous variables (0 = No, 1 = Yes), we rounded upward when there was disagreement (e.g., one shopper observed a feature of the site and the other did not). Where appropriate, we examined the factor structure of the checklist’s domains using Principal Axis Factoring with Varimax rotation [27]. We also estimated each domain’s reliability coefficient (i.e., Cronbach’s alpha for continuous variables or SK-20 for dichotomous variables). Finally, we examined whether site scores varied depending on whether the site offered HIV-only testing (n = 14) or comprehensive HIV/STI testing (n = 29) using χ² and t tests, as appropriate. To reduce Type-I errors in our bivariate analyses, we adjusted our alpha to p ≤ 0.01. For brevity, we note in the Results section only those statistically-significant findings where we observed bivariate differences across testing sites.

Results

Clinic Characteristics

The majority of sites (39 of 43) allowed individuals to schedule a visit over the phone, whereas only ten sites had online appointment capabilities. Most appointments were available by the next business day (see Table 1). On average, visits lasted 45 min.

The majority of the agencies were located near public transportation (e.g., bus route). Overall, shoppers perceived the clinics’ environment to have youth-specific materials, as indicated by the presence of youth-targeted décor and printed materials in the clinic (see Table 1). These two youth indicators were highly correlated across sites (α = 0.95). Less than half of all sites had posters, printed materials, or welcoming symbols aimed at LGBT audiences. We created a LGBT visibility scale using these three items (α = 0.84). Most sites (n = 40) used LGB-inclusive language in their forms; a slightly smaller proportion of sites (n = 36) used transgender-inclusive language on their medical forms.

In their assessment of the clinic environment, shoppers perceived the office staff as friendly and LGBT sensitive, including using LGB-affirming language when speaking to the shoppers. Using these five items (see Table 1) we created a clinic environment scale focused on evaluating the office staff’s overall approachability. Factor analysis scores indicated a one-factor solution explaining 40.36 % of the variance with adequate reliability (α = 0.76). Overall, shoppers perceived that most sites explained and maintained confidentiality during their visits (see Table 1).

Patient-Provider Interactions

During the behavioral screener, less than half of all sites (n = 19) inquired about shoppers’ sexual orientation. Sixty percent of sites (n = 26) inquired about shoppers’ relationship status or experiences with intimate partner violence. HIV-only testing sites were more likely to ascertain partner violence than comprehensive HIV/STI testing sites (see Table 2). We created a relationship context scale based on these three items (α = 0.69).

All agencies offered HIV testing to the mystery shoppers. Among the 33 sites that offered comprehensive HIV/STI testing, 19 sites (44.2 %) offered to do a syphilis test via blood draw, 12 (27.9 %) sites had providers who suggested testing the throat for gonorrhea, and 8 (24.2 %) suggested testing the rectum for chlamydia and gonorrhea. Two sites (6.06 %) suggested swabbing the penis for chlamydia, gonorrhea and trichomoniasis. Four sites (9.3 %) offered to conduct a visual inspection of the penis for signs of herpes and genital warts. Two sites (4.7 %) offered to conduct a visual anal inspection. Most providers (n = 41) explained how shoppers could receive their HIV/STI results, yet more than a third of sites (n = 16) did not ask how shoppers would like to be contacted at follow-up (see Table 2).

To assess satisfaction with providers’ risk-reduction counseling, we created a 5-item scale examining shoppers’ counseling session (α = 0.76). Shoppers’ experience of their counseling session varied, with 40 % of sites (n = 17) offering to set action steps to meet safer sex goals or offering risk reduction options (see Table 2). HIV-only testing sites were more likely to help shoppers set action steps than comprehensive HIV/STI testing sites (see Table 2). Provider recommendations delivered were perceived to be valuable across most sites (82.5 %; n = 33). Regarding safer sex education, we found a small proportion of sites ensured that shoppers knew how to use a condom or identify a condom that would work for them (20.9 %; n = 9), or helped them identify a lubricant (18.6 %; n = 8). HIV-only testing sites were more likely to help shoppers think about condom and lubricants than comprehensive HIV/STI testing sites (see Table 2). These three safer sex education items were used to create a safer sex education scale (α = 0.88).

At the end of the visit, shoppers were asked to evaluate providers’ overall competency with HIV/STIs and LGBTQ health issues. Overall, most shoppers agreed that their providers had been competent in these areas. Factor analysis of these two indicators (see Table 2) explained 47.56 % of the total variance, with satisfactory reliability (α = 0.65). Shoppers were also asked to respond to items inquiring about whether they had any negative interactions with their providers during their testing and counseling session (see Table 2). Factor analysis of these 5-items explained over 74.17 % of the variance, with strong reliability (α = 0.89).

Discussion

Consistent with the National HIV/AIDS Strategy [28], HIV/STI testing efforts are crucial to increase HIV/STI status awareness and improve linkage to treatment and retention in care. Local health departments and community-based organizations (e.g., AIDS Service Organizations) often serve as the point of entry for many YGBM to learn about HIV/STI, safer sex education, and receive HIV/STI testing [29]. These agencies may also be perceived as more trustworthy, LGBT-friendly, and/or accessible than other sources (e.g., private providers); however, at present, there are scarce quantitative data documenting the cultural competency of HIV/STI testing sites and the quality assurance of the services provided.

Building on prior qualitative work focused on the needs of adolescent and young adult LGBT clients [3, 16, 17], we developed and found strong psychometric support for a checklist to ascertain HIV/STI testing sites’ clinic characteristics, as well as YGBM’s experiences during the testing and counseling process. Performance on the checklist suggests that the structural characteristics across sites were adequate. For example, most sites were perceived to be accessible, near public transportation, and had quick turn-around times. Although most HIV/STI testing sites scored high regarding the visible signs of youth-specific materials, less than half of all sites visited had printed information or symbols aimed towards sexual minority clients. Investment in structural changes to the clinic environment may increase the environmental friendliness towards sexual minorities, signal inclusivity, and help to promote a safe space for young gay and bisexual men.

Within the testing session, shoppers’ average experience in these sites was characterized by having providers who seemed knowledgeable about HIV/STIs and LGBT health issues, and who helped them feel comfortable during the exchange. Less than half of all sites, however, ascertained shoppers’ sexual orientation. We speculate that the avoidance of discussions related to sexual orientation may be a strategy designed to focus provider expertise on sexual risk behavior events and thereby allow them to sidestep the tendency to conflate HIV/STI risk with youths’ sexuality. We argue, however, that counselors may benefit from ascertaining YGBM’s sexual orientation, as it may help tailor the testing and counseling session, and help to examine whether sexuality-related stressors are influencing clients’ sexual practices (e.g., successful condom negotiation with partners may be harder for youth with greater homonegativity). In addition, only a third of sites determined relationship status. Moreover, we noted that only half of comprehensive HIV/STI testing sites inquired about experiences of intimate partner violence, as compared to the majority of HIV-only testing sites. Taken together, these findings are particularly troublesome given increasing evidence suggesting that relationship dynamics, including partner and relationship characteristics, as well as partner violence, are associated with risk-taking behaviors among YGBM [30].

The breadth of the content in risk reduction counseling and safer sex education varied between HIV-only sites and comprehensive HIV/STI testing sites. While most shoppers acknowledged that providers in both contexts had helped them explore their motivations for testing, sites providing HIV-only testing were more likely than comprehensive HIV/STI testing sites to discuss actionable changes that would improve shoppers’ safer sex goals. Furthermore, only 20 % of all testing sites offered sex education regarding adequate condom and lubricant use. Among these, HIV-only testing sites were more likely than comprehensive HIV/STI testing sites to engage in these discussions. These findings are consistent with Russell et al. findings [22], and suggest the need to reinvigorate patient-centered counseling and offer sex education within comprehensive HIV/STI testing sites. Specifically, sites may wish to ensure that their testing services do not become so medicalized that they fail to provide risk reduction counseling and safer sex education to YGBM, as they are less likely than heterosexual young men to have received this information during their sex education curriculum [31].

The deployment of secret shoppers to HIV/STI testing sites in Southeast Michigan was a feasible strategy to collect cultural competency and quality assurance data for sites. Caution should be employed, however, in implementing this approach [26, 32]. Shoppers must be trained prior to deployment to ensure that they feel comfortable refusing services [33]. Similarly, we believe it is important to let sites know that shoppers visited their location. In line with this goal, at the end of data collection, we sent each site a letter describing our process and encouraging them to schedule a meeting with us to discuss the shoppers’ experiences at their agency. In these meetings, we offered a packet of personalized results, summarizing how they compared on various quantitative indicators to other sites, and provided feedback from the open-ended portion of the evaluation. Agency staff were eager for feedback; 66 % of the sites requested to meet. Some staff admitted nervousness at receiving results, yet stated they ultimately wanted a chance to improve services. With few exceptions, agencies welcomed suggestions for creating more inclusive environments; however, they noted how funding, space, and hierarchical bureaucratic structures created obstacles for the implementation of best practices. Several agencies noted that the report from the site evaluation would be used to discuss strengths, as well as areas for improvement, with site personnel and as preliminary data when seeking additional funding from public and private donors.

Several limitations are worth noting. Although we sought to diversify the data obtained within each site by having the two shoppers visit each site on different times and days, site evaluations are not reflective of all providers in each location. Second, our study does not seek to be generalizable to all testing sites across the United States, as each region may have a unique set of characteristics that influence the availability and quality of testing sites. Third, although our site evaluation tool had strong psychometric properties across most domains, we observed low internal consistency within our privacy/confidentiality and notification of results domains. It remains unclear whether the low reliability is a function of insufficient specificity in the wording of our evaluation tool or if these indicators do not correlate as strongly as hypothesized. Further testing of our evaluation tool in other settings and with other populations is warranted. Finally, the cross-sectional design of our study limits our ability to make causal inferences about the data reported in this manuscript. Furthermore, we do not know whether the tailored feedback provided to the sites was used to improve the sites’ performance. Future research examining the validity of the tool in predicting YGBM’s likelihood to repeat test and/or seek treatment in certain locations may be warranted.

Implementation of quality assurance systems for HIV/STI testing services may be warranted and may offer opportunities to strengthen the delivery of culturally competent care. Our findings underscore the importance of considering how site characteristics may influence YGBM’s testing motivations and behaviors. Encouraging HIV/STI testing among YMSM within public health campaigns may lose effectiveness if our testing sites provide services that are not culturally competent to this population. Although mystery shopper is a novel strategy in the HIV/STI testing context, we recognize that some agencies and community groups may have challenges implementing this strategy. Nevertheless, our developed indicators may be used by HIV/STI testing sites and/or funders to assess site performance. For example, agencies could include these indicators in their clients’ exit satisfaction surveys and review them quarterly to prioritize areas of need with their staff. Similarly, funders could ask agencies to collect these data and report them in their annual reports, use these data to incentivize sites with evidence of strong care, and/or provide technical assistance to sites that are underperforming. Future research examining the implementation of these systemic strategies and their effect on YGBM’s testing and engagement in care is warranted.