Abstract
Background
HIV severely impacts the transgender communities in the United States, and transgender women have the highest HIV incidence rates among any identified risk group. Guided by formative research with transgender women and by an expert advisory panel of transgender women, we designed a prototype mobile app to promote HIV prevention among transgender women.
Objective
This study aimed to develop and test the usability and acceptability of the prototype Trans Women Connected mobile app.
Methods
We engaged in a 3-phase prototype development process. After conducting formative research about the health needs of this population, we outlined a theory-based app framework and developed three prototype activities (ie, a vision board, a pre-exposure prophylaxis [PrEP] education activity, and an interactive map). We then tested the usability and acceptability of the mobile app and activities with 16 transgender women using pre- and posttests, think-aloud protocols, and open-ended questions.
Results
Participants reported high acceptability for the mobile app; the mean rating across all usability and likability questions was 5.9 out of 7. Service utilization intention, goal setting, and social support increased at posttest compared with pretest. Increases in self-efficacy in finding lesbian, gay, bisexual, transgender, and queer–friendly services; intention to seek online social support; and PrEP knowledge were statistically significant. Participants described the app as attractive and useful and perceived all three activities positively.
Conclusions
This study describes the development and usability and acceptability evaluation of a prototype mobile app designed for and with transgender women for HIV prevention. The usability testing findings provided important insights toward refining and the further development of the Trans Women Connected mobile app. The results suggest that a mobile health intervention can support positive changes. The remaining development and efficacy randomized trial of the Trans Women Connected mobile app is currently underway.
Keywords: transgender persons, HIV, sexual health, minority health, health care disparities, health status disparities
Introduction
According to a 2016 meta-analysis, nearly 1 million adults in the United States are estimated to be transgender [1]. Transgender (or trans, in short) is a term for individuals whose gender expression and/or gender identity do not align with cultural expectations and gender norms associated with their sex assignment at birth [2]. Transgender women in the United States have been severely impacted by HIV [3-8], and more than 25% of them are living with HIV [5]. Transgender women of color are particularly affected; 56% of black/African American transgender women are living with HIV [5]. HIV incidence rates among transgender people are much higher than the national average [9] and highest among any group specifically tracked by the Centers for Disease Control and Prevention (CDC) [10]. Currently, the only transgender-specific program in the CDC’s Effective Interventions and Compendium of Evidence-Based Interventions and Best Practices for HIV [11,12] is an in-person, couples-based intervention designed for transgender women coupled with a cisgender male partner [13].
Internet/electronic health (eHealth) and mobile health (mHealth) modalities may be particularly appropriate for transgender women, many of whom are socially marginalized, who live in areas that do not offer transgender-specific programming and/or may be uncomfortable participating in face-to-face interventions because of confidentiality concerns and fear of stigmatization [5,14-16]. Although empirical data on internet and mobile app use among transgender persons is limited, existing studies show that transgender women consistently use their phones for information gathering, socializing, and making sexual connections [17-19]. These communication practices provide opportunities for engaging transgender women in HIV prevention and sexual health promotion issues. Mobile phone apps also have the capacity to deliver individually tailored content based on motivations for use and app use patterns [20], making them particularly appropriate for addressing the heterogeneous experiences, identities, and needs of transgender women across the life course. Furthermore, although there are very few studies evaluating transgender-specific eHealth or mHealth programs [21], a growing body of research about diverse communities shows that new media intervention programs are acceptable and can be effective in reducing HIV/sexually transmitted infection–related risk behaviors and linking individuals to prevention and care services [22-34]. Given these realities, program developers and practitioners are increasingly calling for transgender-specific internet-, social media-, and mobile-based programs to expand the reach of HIV prevention and health promotion activities [3,35-38].
The purpose of this study was to develop and evaluate the usability and acceptability of a prototype of the Trans Women Connected mobile app that addresses the unique HIV prevention and sexual health needs of transgender women. The goal was to engage transgender women through a strengths-based approach to HIV prevention and sexual health promotion, leveraging the power of social networks to identify and encourage protective factors.
Methods
Overview
We developed the prototype in three phases (see Figure 1). This process occurred in collaboration with a research team, an expert advisory panel (EAP), and a technology team. The EAP consisted of 3 transgender women of color, a Latinx woman, a Filipina woman, and an African American woman, each of whom holds a high-level position in an organization providing services to transgender women. They provided consultation and feedback throughout the prototype design and development process and were compensated as project consultants. The technology team included a technology director, an app developer, a database engineer, and a user experience and graphic designer and was responsible for the design and programming of the prototype. The ETR Institutional Review Board approved the study protocols and provided oversight. In phase one, the research team conducted formative research with potential users of the mobile app to understand the health needs of this population and how an app might support their health and to gather specific suggestions for the development and content of the app. Phase two included prototype design and development of the overall app structure and three activities (ie, a vision board, a pre-exposure prophylaxis [PrEP] education activity, and an interactive map). In phase three, a group of potential users used, rated the usability and acceptability of, and described their experiences with the prototype.
Figure 1.
Prototype development process. PrEP: pre-exposure prophylaxis; STD: sexually transmitted disease.
Phase One: Formative Research
The research team conducted four focus groups and 20 in-depth qualitative interviews with transgender women between the ages of 18 and 59 years living in urban and rural areas in every region of the United States [39]. Briefly, women were recruited through partnerships with community-based organizations, fliers in community spaces and on social media, and word of mouth. Focus group participants received US $100 in compensation; interview participants received US $50. Topics included overall well-being and connectedness, transgender health, sexual health, use of internet and social media, and recommendations for a trans-specific app. Focus groups and interviews were audio-recorded and transcribed. We used the grounded theory open coding methodology and ethnographic methodologies to classify key themes [40,41].
Phase Two: Prototype Design and Development
On the basis of the findings of the formative research, the research team developed a preliminary curriculum outline, which was discussed with and refined by the EAP and, at times, additional transgender women from their networks. The technology team then determined the optimal formats for conveying the curriculum in an engaging manner, in small self-paced segments, with sufficient levels of interactivity, and customization. The technology team next created wireframes and storyboards to mock-up the layout, workflow, and interaction of all app elements. Through an iterative process of review and feedback, the EAP guided the designer to create a final set of imagery and style guide (see Figure 2 for the overall look). Following final approval of the storyboards, the app developer created the prototype framework using Adobe Animate. Using this cross-platform development method, the app could be built for several mobile platforms, including Android and iOS, and would allow the utilization of integrated mobile device (eg, mobile phone and tablet) features such as GPS and camera.
Figure 2.
Welcome screen for the Trans Women Connected mobile app.
Phase Three: Usability Testing
Participants
Two community-based organizations, one in San Francisco, California and one in Portland, Oregon, recruited participants. Inclusion criteria were ages 18 to 55 years, self-identify as a transgender woman, and English speaking. All participants provided written informed consent. Participants received US $100 in compensation.
Procedures and Measures
Using a standard think-aloud protocol, the usability test administrator (TK) asked each participant to complete a series of tasks associated with typical use case scenarios, such as using the menu to locate activities and completing activities. We also objectively measured participants’ task completion success and time to completion and asked them to articulate their questions and thinking process [42,43].
Participants rated the difficulty, appearance, perceived usefulness, and educational value following each key task and after completion of all activities and completed a pre- and posttest that assessed service utilization, goal setting, social support, and HIV PrEP knowledge. We also solicited open-ended comments regarding their perceptions of the app and suggestions to improve it.
Analysis
We conducted descriptive analyses of the ratings and assessed changes from pre- to posttest using nonparametric tests in IBM SPSS version 24 (IBM Corp, Armonk, NY). The research team reviewed open-ended comments and identified key suggestions.
Results
Phase One: Formative Research
Themes About Transgender Health
We identified four main themes from the focus groups and interviews with 57 transgender women (45/57, 79% who were transgender women of color) related to the health of transgender women:
Employment and structural factors play a key role in the health of transgender women.
Transition-related care and mental health care are highly prioritized by transgender women. However, knowledgeable and affirming providers are difficult to find.
The sexual health needs of transgender women are much broader than just HIV and sexually transmitted infections.
Finding community and social support is challenging, but an important part of health for transgender women.
Conceptual Feedback About a Mobile App for Transgender Health
Our formative research found that most participants regularly used social media channels such as Facebook, Instagram, Snapchat, and various social and sexual networking apps to connect with friends, family, romantic/sexual partners, and work opportunities. However, the women also believed that the Trans Women Connected mobile app could fill a gap by providing resources specific to transgender women and an opportunity to connect to other transgender women (see Textbox 1). One Atlanta focus group participant described how this app could be a tool for “lifting the curtain and letting go of baggage.” In particular, participants valued possible app features, such as forums and direct messaging, to enable users to engage with each other and for the possibility of formal mentoring. Participants had questions and concerns about how the app would be structured, including issues of privacy and safety, and the possibility of forums and messaging being used for dating or hooking up. In general, however, participants were positive about the mobile app, and many expressed that it would fulfill an important need in the community.
Suggestions for Trans Women Connected mobile app features and activities.
Give transgender women a voice in the process at every step of development
Images and videos used in the app depict actual transgender women
Space for trans women to connect and share information
Optional and customizable notifications
Customizable privacy settings to allow users to control how their information is displayed
Information about sexual health, including sexual pleasure
Support for healthy relationships and relationship skills
Information to help women locate both trans-specific resources and trans friendly businesses and organizations
Ways to block and report other users
Phase Two: Prototype Development
Guided by the formative research and EAP, the technology team created three prototype interactive activities. The first activity, entitled Create Your Own Vision (Board), provides participants with the opportunity to imagine how they would like their lives to be in eight domains: health, living situation, school/work, sex/love, making a difference, people, free time, and spirituality (see Figure 3). This activity seeks to help transgender women situate their well-being and sexual health within the social determinants of health and structural factors that emerged as the dominant theme of the focus groups and interviews.
Figure 3.
A screenshot of the Create Your Own Vision (Board) prototype activity.
This interactive vision board allows participants to use their existing pictures, take pictures, create doodles/artwork, select quotes, and add their own text or other images to create a vision for each of the eight life domains. These eight collages are merged into a final scrollable collage that represents the user’s vision for their future. The activity also has the user identify a primary goal for their life today, lay out the steps needed to reach that goal, and identify who can help them execute each step. This goal setting activity provides a roadmap to reaching their goal that users can return to anytime. In addition, the vision board can be printed or shared with other app users—an option that all usability testers said they would use.
The second activity—Is PrEP Right for Me or My Partners?—presents a basic overview of PrEP and guides participants through a benefits and risks assessment to help them decide if PrEP might be appropriate for them or their partners (see Figure 4). The activity primarily presents information through a series of videos featuring a charismatic African American transgender woman who is getting ready for and enjoying a night out with friends, including 2 transgender women who tell personal stories about using PrEP (see Figure 5). Interactive elements were included through questions and a poll that allows the user to see what other users are thinking about PrEP and a map that identifies PrEP providers in the users’ area.
Figure 4.
A screenshot of the Is pre-exposure prophylaxis Right for Me or My Partners? prototype activity.
Figure 5.
A screenshot of a video within the Is pre-exposure prophylaxis Right for Me or My Partners? prototype activity.
Responding to the need for helping transgender women find culturally competent and welcoming services, the staff developed the third activity: Interactive Resource Map. The map displays providers/organizations in five service categories: medical, support, educational, employment, and housing/living. On the basis of GPS location, users see icons for each entry and can customize the map to view all or certain categories. When users choose an icon, they see details about the resource including name, contact information, and user ratings and reviews (see Figure 6). In addition, the map allows users to add locations to the map as they identify transgender-friendly providers in their area.
Figure 6.
A screenshot of the Interactive Resource Map prototype activity.
Phase Three: Usability Testing
Participant Characteristics
A total of 16 racially diverse transgender women participated in the usability testing (see Table 1). Participants were aged 19 to 52 years, with an average of 34.5 years, and all were experienced with using mobile devices.
Table 1.
Participant characteristics.
| Characteristics | Values | |
| Location, n (%) | ||
|
|
San Francisco | 11 (69) |
|
|
Portland | 5 (31) |
| Gender identity, n (%) | ||
|
|
Transgender woman | 14 (88) |
|
|
Nonbinary | 2 (13) |
| Racea, n (%) | ||
|
|
American Indian or Alaska Native | 3 (19) |
|
|
Asian | 5 (31) |
|
|
Black or African American | 3 (19) |
|
|
Native Hawaiian or other Pacific Islander | 1 (6) |
|
|
White/Caucasian | 6 (38) |
|
|
Other | 3 (19) |
| Ethnicity, n (%) | ||
|
|
Hispanic or Latinx | 2 (13) |
|
|
Non-Hispanic or Latinx | 14 (88) |
| Age (years), mean (SD) | 34.5 (9.28) | |
| Experience with mobile devices, n (%) | ||
|
|
Very experienced | 14 (88) |
|
|
A lot of experience | 1 (6) |
|
|
Some experience | 1 (6) |
|
|
Not much experience | 0 (0) |
|
|
No experience | 0 (0) |
| Primary mode to connect with transgender women, n (%) | ||
|
|
Online | 0 (0) |
|
|
In person | 7 (44) |
|
|
Both online and in person | 8 (50) |
|
|
Other | 1 (6) |
| Primary mode to meet transgender women, n (%) | ||
|
|
Online | 3 (19) |
|
|
In person | 9 (56) |
|
|
Both online and in person | 4 (25) |
|
|
Other | 0 (0) |
aParticipants could select more than one option.
App Usability
The usability test results demonstrated that participants liked the mobile app and found it highly usable (see Table 2). The mean rating across all usability/likability questions was 5.9 out of 7. The study benchmark was a score of 4 or better for each individual rating component.
Table 2.
Usability and likability ratings of the Trans Women Connected mobile app.
| Variable | Values, mean (SD)a | |
| Main menu | ||
|
|
Likeability of opening screen | 6.25 (1.24) |
|
|
Likeability of menu system | 5.66 (1.81) |
| Create Your Own Vision (Board) | ||
|
|
Likeability of the appearance | 6.28 (1.49) |
|
|
Likeability of the interface | 6.44 (0.73) |
|
|
Likeability of the narrator | 5.80 (1.14) |
| Is PrEPb Right for Me or My Partners? | ||
|
|
Likeability overall | 5.57 (1.60) |
|
|
Likeability of women in the videos | 6.14 (1.03) |
| Interactive Resource Map | ||
|
|
Usefulness | 6.73 (0.59) |
| Overall ratings of Trans Women Connected | ||
|
|
Appearance | 6.38 (1.15) |
|
|
Contents | 6.39 (0.65) |
|
|
Ease of use | 6.25 (1.29) |
|
|
Interest | 6.08 (1.16) |
|
|
Professionally designed | 5.92 (1.38) |
|
|
Enjoyability | 6.25 (1.48) |
|
|
Usefulness | 6.50 (1.24) |
|
|
Educational value | 6.25 (0.87) |
|
|
Likelihood of using | 6.58 (0.68) |
|
|
Likelihood of recommending | 6.62 (1.39) |
a7 represents the most positive or highest rating; the study benchmark was 4.
bPrEP: pre-exposure prophylaxis.
Changes in Service Utilization, Goal Setting, Social Support, and Pre-Exposure Prophylaxis Knowledge
There was a positive trend and increase across multiple measures (eg, perceptions of available social support, service utilization intention, intention to mentor other transgender women on the app, and self-efficacy in discussing PrEP; see Table 3) after using the app. Increases in self-efficacy in finding lesbian, gay, bisexual, transgender, and queer (LGBTQ)–friendly services; intention to seek online social support; and PrEP knowledge were statistically significant.
Table 3.
Self-efficacy, intention, knowledge, goal setting, and social support: pre– and post–usability testing scores.
| Variable | Usability testing scores | P value | |
|
|
Pre | Post |
|
| Self-efficacy in finding LGBTQa-friendly services, mean (SD) | 3.19 (0.75) | 3.69 (0.48) | .01 |
| Likelihood of seeking LGBTQ-friendly services, mean (SD) | 3.69 (0.48) | 3.56 (0.81) | .48 |
| Intention to make medical appointmentb, mean (SD) | 3.88 (0.34) | 3.88 (0.50) | >.99 |
| Created health goals, n (%) | 13 (81) | 13 (81) | >.99 |
| Considered steps to reach goalc, n (%) | 9 (69) | 9 (69) | >.99 |
| Have social support to help reach goal, n (%) | 12 (75) | 14 (88) | .16 |
| Availability of social support, mean (SD) | 2.63 (1.02) | 3.00 (0.97) | .71 |
| Intention to seek online supportb, mean (SD) | 2.56 (1.09) | 3.06 (1.00) | .03 |
| Intention to seek local support servicesb, mean (SD) | 3.38 (0.81) | 3.50 (0.63) | .48 |
| Helpfulness of mentoring from another trans woman, mean (SD) | 4.06 (1.24) | 4.19 (1.28) | .91 |
| Intention to mentor other transgender women on app, mean (SD) | 3.38 (0.89) | 3.56 (0.63) | .48 |
| Can identify friends/family for support, n (%) | 15 (94) | 16 (100) | >.99 |
| PrEPd knowledge, mean (SD) | 2.50 (1.37) | 3.56 (0.63) | .008 |
| Intention to seek more info about PrEPb, mean (SD) | 2.80 (1.21) | 2.81 (1.22) | .58 |
| Intention to discuss PrEP with providerb, mean (SD) | 2.07 (0.80) | 2.00 (0.97) | >.99 |
| Intention to discuss PrEP with partnerb,e, n (%) | 6 (40) | 6 (40) | >.99 |
| Self-efficacy in discussing PrEP, mean (SD) | 3.00 (1.13) | 3.25 (0.86) | .16 |
| Connection to other transgender women, mean (SD) | 6.44 (2.60) | 6.63 (2.33) | .78 |
| Satisfaction with connection to other transgender women, mean (SD) | 5.00 (1.67) | 5.19 (1.83) | .26 |
| Level of social support from other transgender women, mean (SD) | 5.19 (1.87) | 5.21 (1.81) | .72 |
aLGBTQ: lesbian, gay, bisexual, transgender, and queer.
bIn the next 30 days.
cOut of 13 participants.
dPrEP: pre-exposure prophylaxis.
eOut of 15 participants.
Qualitative Findings From Usability Testing
Usability testing participants also provided a number of qualitative comments about their experience using the prototype and suggestions for how it might be improved (see Textbox 2). Overall, this qualitative feedback was positive. As one participant told us, “Helping trans people navigate their health is a win.” As in the quantitative ratings, women expressed that they felt the app was useful and attractive. The welcome screen was described as being friendly and approachable, and many participants listed the esthetics of the app as one of their favorite parts. The women liked all three activities, with the map being a particular favorite. The potential for interactivity and connection with other transgender women was described as another attractive feature. The suggestions provided ranged from broad to very specific; a list of some of the most common or useful suggestions can be seen in Textbox 2.
Suggestions to improve the Trans Women Connected mobile app prototype.
General
More information or instructions on welcome screen and main menu
Tidier appearance/colors
More photos with a variety of transgender women (more representative of the community)
Options for text instead of audio; subtitles for all audio
Change or remove audio when returning to a screen
More black trans women developers
Provide suggestions for donating or volunteering
Create Your Own Vision (Board)
Improve instructions and make navigation clearer
Ability to draw directly on vision board
Ability to add songs, gifs, or other “moving” content
Option to turn off narration
Option to share on social media
Is PrEP Right for Me or My Partners?
Option to pause videos
Shorter videos
Option to view more scientific information
Clarify some information
Interactive Resource Map
Add a search bar and list of locations
Ability to filter user-added locations
Filter or information on Americans with Disabilities Act-accessible locations
Add other types of resources, such as gender-neutral restrooms
Add a filter or new category for free activities
Discussion
Principal Findings
We conducted a 3-phase process to develop a prototype HIV prevention mobile app for and with transgender women. In the first phase, we completed formative research, which indicated that HIV prevention for transgender women must address the larger context of women’s lives. In the second phase, we developed and refined the overall app framework and three prototype activities based on feedback from the EAP. In the third phase, we conducted usability testing. Participants rated the app highly on usability and likeability. There was a significant increase in self-efficacy in finding LGBTQ-friendly services, intention to seek online social support, and PrEP knowledge. Overall, the participants were supportive of the app, with many saying they would use it regularly with no additional incentive beyond the information and opportunity to connect with other transgender women.
Strengths of the Approach
There are a number of strengths of the Trans Women Connected prototype mobile app and research process. The 3-phase development process allowed us to collect formative data about the health priorities of transgender women and incorporate this information into the app prototype. We developed these activities because they addressed issues or requests observed in the formative research. This approach, which is considerably more extensive than the typical app development approach, helps to ensure that the selected prototype activities were addressing real needs of the community. The extensive formative research and the rigor of the analysis in this project, when combined with an iterative usability testing approach, are unique among mobile app developers.
Although researchers have called for digital health interventions to (1) be grounded in behavioral theory [44,45], (2) utilize an in-depth qualitative understanding of the intended population [46], and (3) be iteratively developed with multiple stages of user feedback [47,48], the process has been rarely implemented consistently, with variations in the development process between academic researchers and technology industry developers. Most mHealth interventions designed by those within the industry do not yet incorporate theory-based strategies known to drive changes in health behaviors or undergo systematic testing to demonstrate their effectiveness [45,49,50]; however, they typically employ a rapid iterative development process of the technology. In contrast, interventions developed within an academic environment are more often grounded in behavioral theory [51], but researcher-driven mobile interventions often do not benefit from rapid iterative prototyping and standardized usability testing with iterative feedback [52].
The development of this app integrated the strategies used in both academia and the industry and, in some instances, used methodologies that were more rigorous. For example, rather than use a local convenience sample for focus groups, as is typical of most software and app developers, the project conducted focus groups with transgender women in four regions and interviews with transgender women throughout the United States, with key transgender community stakeholders actively assisting in recruitment activities. By using a community-driven, grounded theory approach, the research team was able to identify and delve deeply into themes that would not have been detected through simply examining focus group transcripts from convenience samples.
Furthermore, input from transgender women was integrated throughout the prototype development in coordination with the EAP and women from their networks. These iterative feedback loops began earlier than typically in app development and included gathering feedback on colors and image representations of transgender women and several other functional and aesthetic elements of the app before moving on to feedback on the app interface and activity content, function, and appearance. The inclusion of members of the LGBTQ community as part of the development process extended beyond formative research into voice talent/narration, video development, and design, with the designer being an LGBTQ-identified woman of color, the narrators being transgender women of color, and all video participants being transgender women. These multiple forms of community engagement not only resulted in a highly tailored app but also further provide a foundation for dissemination and active utilization once the Trans Women Connected mobile app is finalized. As suggested by other researchers, community engagement increases the likelihood the knowledge gained and the intervention, in this case an app, will benefit the community [53-55]. We recognize and appreciate the importance of the community’s priorities and solutions, which enhances the relevance and use of data and quality and validity of research [56-58].
Additional strengths of the usability testing were the multiple types of data and an evaluation of impact. We gathered both quantitative and qualitative data about the experience of using the prototype mobile app; the quantitative data allowed us to verify that we were meeting numeric targets for usability and acceptability (as is typical for usability testing), whereas the qualitative data gave participants the opportunity to provide more in-depth feedback and suggestions. However, measuring the impact of the prototype on knowledge, attitudes, and intentions during the development process rather than when the app is completed goes considerably beyond standard usability testing, which is typically limited only to identifying problems within the user experience [59]. This approach helps determine whether the learning and engagement activities are performing as intended before building out the entire app.
Finally, a central strength of this mobile app is the emphasis on addressing HIV holistically using a strengths-based perspective. This approach recognizes the resilience of the transgender community and the social and structural barriers that transgender women face [60]. Developing an effective intervention requires a comprehensive approach that intentionally targets community strengths and challenges [60-63]. Our formative research confirmed that attention to the larger context of trans women’s lives was key to HIV prevention [14], [61,64,65], and this understanding guided our development process.
Limitations
A few limitations of our study need to be considered. The usability testing sample was limited to 16 participants in two different cities in the West Coast. Research has determined that 7 to 10 usability participants will uncover 80% to 90% of usability problems [66,67], and we were able to test the mobile app with a diverse group of 16 participants. This sample allowed the examination of some of the different experiences and identities of transgender women. We likely identified functional problems with the app and the app was well received as usable and acceptable; however, it is possible that more geographically diverse participants or participants from rural areas would have qualitatively experienced the app differently.
The inclusion of pre- and posttest knowledge and intention measures during a usability test is unusual and presents some limitations. Although significant increases in knowledge and intent were found on some variables, the sample size, although adequate for usability testing, is small for such testing and again may not represent the effect that would have been found with a more diverse or larger sample. However, the results are promising given that there were positive changes in key measures after viewing only three prototype activities, and this would seem to bode well for the efficacy of the complete mobile program. For several measures, there was no significant increase between pre- and posttest scores; we may have observed a ceiling effect. Given the short duration of app use, we were unable to measure behavior change. However, we measured antecedents of behavior change according to behavior change theories, and in the full trial when participants have a longer period to use the mobile app, we expect to observe increases in these antecedents and more protective behaviors.
A final limitation of the project was that all members of the research team identify as cisgender. Although the input of transgender women was included throughout the development process, no transgender individuals participated directly in the interpretation and analysis of data. As a result, there may be aspects of the data that were missed or misinterpreted because of the absence of this perspective.
Next Steps
Given the positive results from the prototype development and usability testing, our next steps are to complete the development of the mobile app, including the storyboards and curriculum/content for the remaining activities, and conduct a cluster randomized controlled trial to assess the effectiveness of the Trans Women Connected mobile app. The expansion of the app to include additional activities will be based on a theoretical framework of gender affirmation, resilience, and cognitive behavioral theory and will be implemented using an agile development process with iterative testing. Through the entire process, we will continue to work with an expanded expert and community advisory group of transgender women who will provide feedback on the app; they will review materials to ensure the content reflects their experiences and those of their community. The project team, in collaboration with the expert advisors, will identify the most essential topics, which will then be reviewed and revised through an iterative process of refinement. A newly formed community advisory board, consisting of 10 geographically and racially diverse transgender women, will provide feedback on the activity plans and storyboards. The project team will then revise the documents, with the process continuing until the content is approved by the community advisors. In addition, the community advisory board will test and review completed interactive activities throughout the project.
Conclusions
Strong formative research, followed by an iterative review from members of the intended audience, resulted in a usable and acceptable mobile app that has been well received by transgender women. Usability testing findings provided important insights toward refining and the further development of the Trans Women Connected mobile app. The results of this study suggest that an mHealth intervention can address critical structural factors that shape transgender women’s lives and support positive changes in the knowledge and attitudes about social connection and health access that are antecedents to increasing protective and health promotion behaviors.
Acknowledgments
The authors would like to extend their gratitude for the time and effort of the transgender women who provided input and feedback on their app in the formative research interviews and focus groups, as members of the EAP, and in usability testing. They would also like to thank the community-based organizations that supported this project. This project was supported by grant R43MD012279 from the National Institute on Minority Health and Health Disparities and K12HS022981 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Minority Health and Health Disparities and the Agency for Healthcare Research and Quality.
Abbreviations
- CDC
Centers for Disease Control and Prevention
- EAP
expert advisory panel
- eHealth
electronic health
- LGBTQ
lesbian, gay, bisexual, transgender, and queer
- mHealth
mobile health
- PrEP
pre-exposure prophylaxis
Footnotes
Conflicts of Interest: None declared.
References
- 1.Institute of Medicine. Board on the Health of Select Populations. Transgender Health Issues and Research Gaps and Opportunities Committee on Lesbian, Gay, Bisexual . The Health Of Lesbian, Gay, Bisexual, And Transgender People: Building A Foundation For Better Understanding. Washington DC: The National Academies Press; 2011. [PubMed] [Google Scholar]
- 2.Meerwijk EL, Sevelius JM. Transgender population size in the United States: a meta-regression of population-based probability samples. Am J Public Health. 2017 Feb;107(2):e1–8. doi: 10.2105/AJPH.2016.303578. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Centers for Disease Control and Prevention. 2019. [2019-08-11]. HIV And Transgender People http://www.cdc.gov/hiv/group/gender/transgender/
- 4.Clements-Nolle K, Marx R, Guzman R, Katz M. HIV prevalence, risk behaviors, health care use, and mental health status of transgender persons: implications for public health intervention. Am J Public Health. 2001 Jun;91(6):915–21. doi: 10.2105/ajph.91.6.915. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Herbst JH, Jacobs ED, Finlayson TJ, McKleroy VS, Neumann MS, Crepaz N, HIV/AIDS Prevention Research Synthesis Team Estimating HIV prevalence and risk behaviors of transgender persons in the United States: a systematic review. AIDS Behav. 2008 Jan;12(1):1–17. doi: 10.1007/s10461-007-9299-3. [DOI] [PubMed] [Google Scholar]
- 6.Operario D, Soma T, Underhill K. Sex work and HIV status among transgender women: systematic review and meta-analysis. J Acquir Immune Defic Syndr. 2008 May 1;48(1):97–103. doi: 10.1097/QAI.0b013e31816e3971. [DOI] [PubMed] [Google Scholar]
- 7.Poteat T, Scheim A, Xavier J, Reisner S, Baral S. Global epidemiology of HIV infection and related syndemics affecting transgender people. J Acquir Immune Defic Syndr. 2016 Aug 15;72(Suppl 3):S210–9. doi: 10.1097/QAI.0000000000001087. http://europepmc.org/abstract/MED/27429185. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Reback CJ, Fletcher JB. HIV prevalence, substance use, and sexual risk behaviors among transgender women recruited through outreach. AIDS Behav. 2014 Jul;18(7):1359–67. doi: 10.1007/s10461-013-0657-z. http://europepmc.org/abstract/MED/24287786. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Grant JM, Mottet LA, Tanis J, Harrison J, Herman JL, Keisling M. National LGBTQ Task Force. Washington, DC: National Center for Transgender Equality and National Gay and Lesbian Task Force; 2011. [2019-08-12]. Injustice at Every Turn: A Report of the National Transgender Discrimination Survey https://www.thetaskforce.org/wp-content/uploads/2019/07/ntds_full.pdf. [Google Scholar]
- 10.Centers for Disease Control and Prevention. 2014. [2019-08-12]. CDC-Funded HIV Testing: United States, Puerto Rico, and US Virgin Islands, 2014 https://www.cdc.gov/hiv/pdf/library/reports/cdc-hiv-funded-testing-us-puerto-rico-2014.pdf.
- 11.Centers for Disease Control and Prevention. 2019. [2019-08-12]. Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention https://www.cdc.gov/hiv/research/interventionresearch/compendium/rr/index.html.
- 12.Centers for Disease Control and Prevention. 2019. [2019-08-12]. Effective Interventions: HIV Prevention That Works https://www.cdc.gov/hiv/effective-interventions/index.html.
- 13.Reisner SL, Menino D, Leung K, Gamarel KE. 'Unspoken Agreements': Perceived Acceptability of Couples HIV Testing and Counseling (CHTC) among cisgender men with transgender women partners. AIDS Behav. 2019 Feb;23(2):366–74. doi: 10.1007/s10461-018-2198-y. [DOI] [PubMed] [Google Scholar]
- 14.Hines DD, Ryan M. It's not just about condoms and sex: using syndemic theory to examine social risks of HIV among transgender women. In: Wright ER, Carnes N, editors. Understanding the HIV/AIDS Epidemic in the United States. Social Disparities in Health and Health Care. Cham, Switzerland: Springer; 2016. pp. 99–130. [Google Scholar]
- 15.Andrasik MP, Yoon R, Mooney J, Broder G, Bolton M, Votto T, Davis-Vogel A, HVTN 505 study team. NIAID HIV Vaccine Trials Network Exploring barriers and facilitators to participation of male-to-female transgender persons in preventive HIV vaccine clinical trials. Prev Sci. 2014 Jun;15(3):268–76. doi: 10.1007/s11121-013-0371-0. http://europepmc.org/abstract/MED/23446435. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kosenko K, Rintamaki L, Raney S, Maness K. Transgender patient perceptions of stigma in health care contexts. Med Care. 2013 Sep;51(9):819–22. doi: 10.1097/MLR.0b013e31829fa90d. [DOI] [PubMed] [Google Scholar]
- 17.Benotsch EG, Zimmerman RS, Cathers L, Heck T, McNulty S, Pierce J, Perrin PB, Snipes DJ. Use of the internet to meet sexual partners, sexual risk behavior, and mental health in transgender adults. Arch Sex Behav. 2016 Apr;45(3):597–605. doi: 10.1007/s10508-014-0432-x. [DOI] [PubMed] [Google Scholar]
- 18.Krueger EA, Young SD. Twitter: a novel tool for studying the health and social needs of transgender communities. JMIR Ment Health. 2015;2(2):pii: e16. doi: 10.2196/mental.4113. https://mental.jmir.org/2015/2// [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Young SD, Jaganath D. Online social networking for HIV education and prevention: a mixed-methods analysis. Sex Transm Dis. 2013 Feb;40(2):162–7. doi: 10.1097/OLQ.0b013e318278bd12. http://europepmc.org/abstract/MED/23324979. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health smartphone apps: review and evidence-based recommendations for future developments. JMIR Ment Health. 2016 Mar 1;3(1):e7. doi: 10.2196/mental.4984. https://mental.jmir.org/2016/1/e7/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Rhodes SD, McCoy TP, Tanner AE, Stowers J, Bachmann LH, Nguyen AL, Ross MW. Using social media to increase HIV testing among gay and bisexual men, other men who have sex with men, and transgender persons: outcomes from a randomized community trial. Clin Infect Dis. 2016 Jun 1;62(11):1450–3. doi: 10.1093/cid/ciw127. http://europepmc.org/abstract/MED/26980878. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Lightfoot M, Comulada WS, Stover G. Computerized HIV preventive intervention for adolescents: indications of efficacy. Am J Public Health. 2007 Jun;97(6):1027–30. doi: 10.2105/AJPH.2005.072652. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Muessig KE, Nekkanti M, Bauermeister J, Bull S, Hightow-Weidman LB. A systematic review of recent smartphone, Internet and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep. 2015 Mar;12(1):173–90. doi: 10.1007/s11904-014-0239-3. http://europepmc.org/abstract/MED/25626718. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Card JJ, Kuhn T, Solomon J, Benner TA, Wingood GM, DiClemente RJ. Translating an effective group-based HIV prevention program to a program delivered primarily by a computer: methods and outcomes. AIDS Educ Prev. 2011 Apr;23(2):159–74. doi: 10.1521/aeap.2011.23.2.159. [DOI] [PubMed] [Google Scholar]
- 25.Catalani C, Philbrick W, Fraser H, Mechael P, Israelski DM. mHealth for HIV treatment & prevention: a systematic review of the literature. Open AIDS J. 2013;7:17–41. doi: 10.2174/1874613620130812003. http://europepmc.org/abstract/MED/24133558. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Klein CH, Kuhn T, Huxley D, Kennel J, Withers E, Lomonaco CG. Preliminary findings of a technology-delivered sexual health promotion program for black men who have sex with men: quasi-experimental outcome study. JMIR Public Health Surveill. 2017 Oct 24;3(4):e78. doi: 10.2196/publichealth.7933. https://publichealth.jmir.org/2017/4/e78/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Noar SM. Computer technology-based interventions in HIV prevention: state of the evidence and future directions for research. AIDS Care. 2011 May;23(5):525–33. doi: 10.1080/09540121.2010.516349. http://europepmc.org/abstract/MED/21287420. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Noar SM, Willoughby JF. eHealth interventions for HIV prevention. AIDS Care. 2012;24(8):945–52. doi: 10.1080/09540121.2012.668167. http://europepmc.org/abstract/MED/22519523. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Schnall R, Travers J, Rojas M, Carballo-Diéguez A. eHealth interventions for HIV prevention in high-risk men who have sex with men: a systematic review. J Med Internet Res. 2014 May 26;16(5):e134. doi: 10.2196/jmir.3393. https://www.jmir.org/2014/5/e134/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Swanton R, Allom V, Mullan B. A meta-analysis of the effect of new-media interventions on sexual-health behaviours. Sex Transm Infect. 2015 Feb;91(1):14–20. doi: 10.1136/sextrans-2014-051743. [DOI] [PubMed] [Google Scholar]
- 31.Conserve DF, Jennings L, Aguiar C, Shin G, Handler L, Maman S. Systematic review of mobile health behavioural interventions to improve uptake of HIV testing for vulnerable and key populations. J Telemed Telecare. 2017 Feb;23(2):347–59. doi: 10.1177/1357633X16639186. http://europepmc.org/abstract/MED/27056905. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Fuchs JD, Stojanovski K, Vittinghoff E, McMahan VM, Hosek SG, Amico KR, Kouyate A, Gilmore HJ, Buchbinder SP, Lester RT, Grant RM, Liu AY. A mobile health strategy to support adherence to antiretroviral preexposure prophylaxis. AIDS Patient Care STDS. 2018 Mar;32(3):104–11. doi: 10.1089/apc.2017.0255. http://europepmc.org/abstract/MED/29565183. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Dillingham R, Ingersoll K, Flickinger TE, Waldman AL, Grabowski M, Laurence C, Wispelwey E, Reynolds G, Conaway M, Cohn WF. PositiveLinks: a mobile health intervention for retention in HIV care and clinical outcomes with 12-month follow-up. AIDS Patient Care STDS. 2018 Jun;32(6):241–50. doi: 10.1089/apc.2017.0303. http://europepmc.org/abstract/MED/29851504. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Sun CJ, Stowers J, Miller C, Bachmann LH, Rhodes SD. Acceptability and feasibility of using established geosocial and sexual networking mobile applications to promote HIV and STD testing among men who have sex with men. AIDS Behav. 2015 Mar;19(3):543–52. doi: 10.1007/s10461-014-0942-5. http://europepmc.org/abstract/MED/25381563. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Arayasirikul S, Wilson EC, Raymond HF. Examining the effects of transphobic discrimination and race on HIV risk among transwomen in San Francisco. AIDS Behav. 2017 Sep;21(9):2628–33. doi: 10.1007/s10461-017-1728-3. http://europepmc.org/abstract/MED/28220311. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Garofalo R, Kuhns LM, Reisner SL, Mimiaga MJ. Behavioral interventions to prevent HIV transmission and acquisition for transgender women: a critical review. J Acquir Immune Defic Syndr. 2016 Aug 15;72(Suppl 3):S220–5. doi: 10.1097/QAI.0000000000001084. http://europepmc.org/abstract/MED/27429186. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Horvath KJ, Iantaffi A, Swinburne-Romine R, Bockting W. A comparison of mental health, substance use, and sexual risk behaviors between rural and non-rural transgender persons. J Homosex. 2014;61(8):1117–30. doi: 10.1080/00918369.2014.872502. http://europepmc.org/abstract/MED/24380580. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.LeGrand S, Muessig KE, Pike EC, Baltierra N, Hightow-Weidman LB. If you build it will they come? Addressing social isolation within a technology-based HIV intervention for young black men who have sex with men. AIDS Care. 2014;26(9):1194–200. doi: 10.1080/09540121.2014.894608. http://europepmc.org/abstract/MED/24617609. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Sun CJ, Anderson KM, Mayer L, Kuhn T, Klein CH. Findings from formative research to develop a strength-based HIV prevention and sexual health promotion mHealth intervention for transgender women. Transgend Health. 2019;4(1):350–8. doi: 10.1089/trgh.2019.0032. http://europepmc.org/abstract/MED/32042925. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Schensul JJ, LeCompte MD. Essential Ethnographic Methods: A Mixed-Methods Approach. Second Edition. Lanham, MD: AltaMira Press; 2012. [Google Scholar]
- 41.LeCompte MD, Schensul JJ. Analysis and Interpretation of Ethnographic Data: A Mixed-Methods Approach. Second Edition. Lanham, MD: AltaMira Press; 2012. [Google Scholar]
- 42.Nielsen J. Usability Engineering. San Diego, CA: Academic Press; 1993. [Google Scholar]
- 43.Dumas JS, Redish JC. A Practical Guide to Usability Testing. Second edition. Exeter, UK: Intellect; 1999. [Google Scholar]
- 44.Azar KM, Lesser LI, Laing BY, Stephens J, Aurora MS, Burke LE, Palaniappan LP. Mobile applications for weight management: theory-based content analysis. Am J Prev Med. 2013 Nov;45(5):583–9. doi: 10.1016/j.amepre.2013.07.005. [DOI] [PubMed] [Google Scholar]
- 45.Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task? Transl Behav Med. 2011 Mar;1(1):53–71. doi: 10.1007/s13142-011-0021-7. http://europepmc.org/abstract/MED/21796270. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res. 2015 Jan 30;17(1):e30. doi: 10.2196/jmir.4055. https://www.jmir.org/2015/1/e30/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Tate EB, Spruijt-Metz D, O'Reilly G, Jordan-Marsh M, Gotsis M, Pentz MA, Dunton GF. mHealth approaches to child obesity prevention: successes, unique challenges, and next directions. Transl Behav Med. 2013 Dec;3(4):406–15. doi: 10.1007/s13142-013-0222-3. http://europepmc.org/abstract/MED/24294329. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Hekler EB, King AC, Banerjee B, Robinson T, Alonso M, Cirimele J, Buman MP, Chen F. A Case Study of BSUED: Behavioral Science-informed User Experience Design. Proceedings of the Conference on Human Factors in Computing Systems; CHI'11; May 7-12, 2011; Vancouver, BC, Canada. New York, NY: Association for Computing Machinery; 2011. http://v1.personalinformatics.org/docs/chi2011/hekler.pdf. [Google Scholar]
- 49.Pagoto S, Schneider K, Jojic M, DeBiasse M, Mann D. Evidence-based strategies in weight-loss mobile apps. Am J Prev Med. 2013 Nov;45(5):576–82. doi: 10.1016/j.amepre.2013.04.025. [DOI] [PubMed] [Google Scholar]
- 50.Mummah SA, Robinson TN, King AC, Gardner CD, Sutton S. IDEAS (Integrate, Design, Assess, and Share): a framework and toolkit of strategies for the development of more effective digital interventions to change health behavior. J Med Internet Res. 2016 Dec 16;18(12):e317. doi: 10.2196/jmir.5927. https://www.jmir.org/2016/12/e317/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Payne HE, Lister C, West JH, Bernhardt JM. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR Mhealth Uhealth. 2015 Feb 26;3(1):e20. doi: 10.2196/mhealth.3335. https://mhealth.jmir.org/2015/1/e20/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Pagoto S, Bennett GG. How behavioral science can advance digital health. Transl Behav Med. 2013 Sep;3(3):271–6. doi: 10.1007/s13142-013-0234-z. http://europepmc.org/abstract/MED/24073178. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.O'Mara-Eves A, Brunton G, Oliver S, Kavanagh J, Jamal F, Thomas J. The effectiveness of community engagement in public health interventions for disadvantaged groups: a meta-analysis. BMC Public Health. 2015 Feb 12;15:129. doi: 10.1186/s12889-015-1352-y. https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-015-1352-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Cyril S, Smith BJ, Possamai-Inesedy A, Renzaho AM. Exploring the role of community engagement in improving the health of disadvantaged populations: a systematic review. Glob Health Action. 2015;8:29842. doi: 10.3402/gha.v8.29842. http://europepmc.org/abstract/MED/26689460. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Centers for Disease Control and Prevention (US) The Principles of Community Engagement. Second Edition. Bethesda, MD: National Institutes of Health; 2011. [Google Scholar]
- 56.Cargo M, Mercer SL. The value and challenges of participatory research: strengthening its practice. Annu Rev Public Health. 2008;29:325–50. doi: 10.1146/annurev.publhealth.29.091307.083824. [DOI] [PubMed] [Google Scholar]
- 57.Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19:173–202. doi: 10.1146/annurev.publhealth.19.1.173. [DOI] [PubMed] [Google Scholar]
- 58.Dankwa-Mullan I, Rhee KB, Williams K, Sanchez I, Sy FS, Stinson N, Ruffin J. The science of eliminating health disparities: summary and analysis of the NIH summit recommendations. Am J Public Health. 2010 Apr 1;100(Suppl 1):S12–8. doi: 10.2105/AJPH.2010.191619. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Nielsen J. Usability Inspection Methods. Proceedings of the Conference Companion on Human Factors in Computing Systems; CHI'94; April 24 - 28, 1994; Massachusetts, Boston, USA. New York, NY: Association for Computing Machinery; 1994. pp. 413–4. [DOI] [Google Scholar]
- 60.Gahagan J, Colpitts E. Understanding and measuring LGBTQ pathways to health: a scoping review of strengths-based health promotion approaches in LGBTQ health research. J Homosex. 2017;64(1):95–121. doi: 10.1080/00918369.2016.1172893. [DOI] [PubMed] [Google Scholar]
- 61.Operario D, Nemoto T. HIV in transgender communities: syndemic dynamics and a need for multicomponent interventions. J Acquir Immune Defic Syndr. 2010 Dec;55(Suppl 2):S91–3. doi: 10.1097/QAI.0b013e3181fbc9ec. http://europepmc.org/abstract/MED/21406995. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 62.Cahill S, Valadéz R, Ibarrola S. Community-based HIV prevention interventions that combat anti-gay stigma for men who have sex with men and for transgender women. J Public Health Policy. 2013 Jan;34(1):69–81. doi: 10.1057/jphp.2012.59. [DOI] [PubMed] [Google Scholar]
- 63.Reback CJ, Ferlito D, Kisler KA, Fletcher JB. Recruiting, linking, and retaining high-risk transgender women into HIV prevention and care services: an overview of barriers, strategies, and lessons learned. Int J Transgend. 2015;16(4):209–21. doi: 10.1080/15532739.2015.1081085. http://europepmc.org/abstract/MED/27110227. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Brennan J, Kuhns LM, Johnson AK, Belzer M, Wilson EC, Garofalo R, Adolescent Medicine Trials Network for HIV/AIDS Interventions Syndemic theory and HIV-related risk among young transgender women: the role of multiple, co-occurring health problems and social marginalization. Am J Public Health. 2012 Sep;102(9):1751–7. doi: 10.2105/AJPH.2011.300433. http://europepmc.org/abstract/MED/22873480. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Mayer KH, Grinsztejn B, El-Sadr WM. Transgender people and HIV prevention: what we know and what we need to know, a call to action. J Acquir Immune Defic Syndr. 2016 Aug 15;72(Suppl 3):S207–9. doi: 10.1097/QAI.0000000000001086. http://europepmc.org/abstract/MED/27429184. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 66.Hwang W, Salvendy G. Number of people required for usability evaluation. Commun ACM. 2010;53(5):130–3. doi: 10.1145/1735223.1735255. [DOI] [Google Scholar]
- 67.Macefield R. How to specify the participant group size for usability studies: a practitioner's guide. J Usability Stud. 2009;5(1):34–45. http://uxpajournal.org/wp-content/uploads/sites/8/pdf/JUS_Macefield_Nov2009.pdf. [Google Scholar]