Crowdsourcing methods to enhance HIV and sexual health services: A scoping review and qualitative synthesis

Abstract

Background:

Crowdsourcing is increasingly used to improve community engagement in HIV and sexual health research. In this scoping review, we reviewed studies using crowdsourcing approaches in HIV and sexual health research to identify strengths, opportunities for expansion, and limitations of such approaches.

Methods:

We searched CINAHL, Web of Science, Embase and PubMed. Studies were included if they involved crowdsourcing activities, were in the field of HIV or sexual health, and described the methodology in sufficient detail. We conducted a qualitative synthesis of eligible articles.

Results:

Our search strategy yielded 431 non-duplicate articles. After screening, 16 articles met the inclusion criteria, including four publications that described research from high-income countries, seven from middle-income countries, one from a low-income country and four that had a global focus. There were four categories of crowdsourcing: open contests, hackathons, open forums, and incident reporting systems. We identified common phases for data acquisition and dissemination: 1) preparation; 2) problem framing and crowd solicitation; 3) judging submissions; and 4) sharing selected submissions. Strengths of using crowdsourcing approaches include greater innovation due to crowd heterogeneity, encouragement of multisectoral collaboration, empowerment of vulnerable populations, cost-effectiveness, and relevance to local cultures and styles. Weaknesses among some methods include reliance on the internet, temporal transience, and difficulty in sustaining long-term community engagement.

Conclusion:

Crowdsourcing may be useful for HIV implementation research. Further research on crowdsourcing related to HIV and sexual health is needed.

INTRODUCTION

The World Health Organization recognizes community engagement as a critical element in the response to the HIV pandemic.¹ Community-driven approaches allow stakeholders to identify problems, generate solutions, and share responsibility for implementing interventions. Crowdsourcing, the process of aggregating a group of non-experts and experts in order to solve a problem, is increasingly used in public health.² Crowdsourcing is based on the principle of collective wisdom and open access. Collective wisdom suggests that in some contexts, crowds as a whole may out-perform the most experienced individuals within them.³ Collective wisdom has been used in political science, philosophy, social science, business and medicine.³

Programs that promote HIV prevention and sexual health may benefit from crowdsourcing, as such approaches could bring large groups of people together in new and creative ways to find culturally appropriate solutions to problems. Allowing a diversity of voices to shape health interventions may guard against group-think and limit less effective top-down, homogeneous interventions.⁴ Crowdsourcing approaches may also save time and costs, and could speed up finding innovative solutions.⁵ Examples of the ways in which crowdsourcing have been used in HIV prevention include logos for an AIDS conference,⁶ videos to increase HIV testing,⁷ and campaigns for promoting HIV testing services.⁸ Crowdsourcing approaches have been utilized in research for key populations, including in sexual health campaigns for youth⁶ and men who have sex with men (MSM).^7-9

To date, there are no crowdsourcing reviews focused on HIV and sexual health, despite the increasing use of this approach in these fields.^4,5,10-12 The purpose of this study is to conduct a qualitative synthesis of existing evidence on the use of crowdsourcing approaches in HIV and sexual health promotion, and to evaluate the advantages and disadvantages of using crowdsourcing.

METHODS

We used Arksey and O’Malley’s framework for conducting this scoping review.¹³ Scoping studies summarize key evidence on a topic, but do not undergo the process of a formal systematic review. We established a list of guiding questions for this review:

1. How did authors define crowdsourcing?

2. What was the purpose and scale of the crowdsourcing approach?

3. What were the types of crowdsourcing approaches?

4. What were the phases or structure of crowdsourcing approaches?

5. What were the strengths and weaknesses of crowdsourcing approaches?

We searched CINAHL, Web of Science, Embase and PubMed on 16 July 2018, identifying papers published that reported crowdsourcing activities related to HIV and sexual health. The literature search included Medical Subject Headings (MeSH), Emtree headings, and related text and keyword searches. Search terms included sexual health-related terms such as HIV, sexually transmitted infection(s), sexual behavior, or condom use. Eligibility criteria included the following: study design/contents involved crowdsourcing activities; study topics were in the field of HIV or sexual health; and methodology was described in sufficient detail. A full search strategy is included in Appendix A. We used PRISMA-ScR checklist (Appendix B) to report this review. Two members of the research team, JJO and WD, independently reviewed manuscript titles and abstracts according to the eligibility criteria. After screening the titles and abstracts, eligibility of relevant citations were further reviewed independently by the two researchers. Discrepancies and uncertainties were discussed with the research group until agreement was achieved. Eligible full articles were included for qualitative synthesis.

RESULTS

Our search strategy yielded 431 non-duplicate articles and 48 manuscripts were fully reviewed. Among the 48 citations, 32 were excluded: 10 had insufficient detail on methods, 16 either did not use crowdsourcing or were not about sexual health, three were duplicates, two were study protocols, and one was a post-hoc analysis of existing ideas from social media platforms. We identified 16 full articles for qualitative synthesis. Table 1 summarizes the aims, methods and participants in the included studies. Four publications reported research from high-income countries (HIC),^9,14-16 seven from middle-income countries (MIC),^7,17-23, one from a low-income country (LIC)²⁴ and four had a global focus.^4,6,20,25 They focused on HIV,^{6,7,9,14,18,19,21,24,25} sexual health promotion,^{15-17,20,22,23} and sexual violence prevention.^15-17 A total of 11 used or evaluated crowdsourcing contests. The geographic scale of crowdsourcing activities included citywide,^{9,14,19,21,22} nationwide,^{7,15-18,20,23} regional²⁴ and global.^4,6,20,25

Table 1.

Summary of the aims, methods and participants in studies included in the review.

Author	Publication year	Geographic location	Aim	Category of crowdsourcing	Crowd contribution
Hildebrand M et al	2013	Global	To develop a strategy for how to better engage young people in decision-making processes on AIDS	Open contest	Young people aged 15–29 from 79 countries. Shared opinions about young people and HIV, including young people living with HIV, access to services, and the legal and social environment, among others.
Mathews A et al	2017	US	To evaluate the feasibility of using a crowdsourcing contest to promote HIV cure research community engagement	Open contest	Submit entries about ‘What does an HIV cure mean to you?’ in the form of videos, images and audio.
Mathews A et al	2018	US	To examine the feasibility of using crowdsourcing contests as a method of HIV cure research engagement	Open contest	Community members participated in focus-group discussions, shared opinions about their acceptance of a crowdsourcing contest, developed a contest name, logo and hashtag.
Ong JJ et al	2017	Global	To evaluate a crowdsourced logo contest as part of an international HIV conference	Open contest	Participants designed and submitted a conference logo.
Pan SW et al	2017	Global	To evaluate the effectiveness and cost of health-focused innovation design contests	Open contest	Contributed and submitted entries
Peuchaud S	2014	Egypt	To describe a case study of how social media activists have harnessed the power of Facebook, Twitter and mobile phone networks to address sexual harassment	Incident reporting system	Individuals who experienced or witnessed sexual harassment reported the incident to the system
Rassenhofer M et al	2015	Germany	To discuss the feasibility and value of using a critical incident reporting system sponsored by the institution responsible versus one sponsored by an independent governmental body	Incident reporting system	Child sexual abuse survivors, confidants or acquaintances of victims, offenders, and people who were concerned about the issue reported the incident or shared thoughts about the topic to the system
Rassenhofer M et al	2013	Germany	To analyze the experiences of now-adult victims of past sexual abuse and make recommendations for policy changes and legislations	Incident reporting system	Child sexual abuse survivors, confidants or acquaintances of victims, offenders, and people who were concerned about the issue reported the incident or shared thoughts about the topic to the system
Seebregts CJ	2009	Africa	To develop an open source electronic medical record application, to provide regional support for application implementations	Open forum	The medical record application implementers and developers communicated problems and solutions on the platform
Tang W et al	2016	China	To compare first-time HIV testing rates among MSM and transgender individuals who received a crowdsourced or a health marketing HIV test promotion video	Open contest	The crowd contributed and submitted HIV test promotion videos
Tang W et al	2018	China	To evaluate the effect of a crowdsourced HIV intervention on HIV testing uptake among MSM	Open contest and hackathon	The crowd contributed and submitted HIV test promotion images, texts, and developed an intervention package for a health campaign.
Tucker JD et al	2018	China	To describe a crowdsourcing designathon, summarize outputs and discuss implications for public health	Hackathon	CDC staff, a local gay community member, and other participants developed an intervention package
Tucker JD et al	2018	Global	To describe crowdsourcing contests by stage, examines ethical problems at each stage, and proposes potential ways of mitigating risk	Open contest	The crowd contributed and submitted entries on relevant topics
Zhang A et al	2018	China	To examine text entries submitted to a crowdsourcing contest on the meaning of an HIV cure	Open contest	Men who have sex with men, people who injected drugs, people living with HIV, and local residents contributed and submitted text entries.
Zhang Y et al	2015	China	To describe creative contributory contests to promote sexual health and provide guidance for designing creative contributory contests	Open contest	Local populations, including MSM, university students, graphic designers and artists contributed and submitted entries on relevant topics
Zhang W et al	2017	China	To increase sexual health awareness among youth using an innovation contest and to evaluate community engagement in the contest	Open contest	Individuals contributed and submitted images promoting sexual health among young people.

Definition of crowdsourcing

Thirteen studies clearly defined their approaches as crowdsourcing.^{6,7,9,12,14,17-23,25} Three studies did not have an explicit definition of crowdsourcing in their study, yet their approach was consistent with the principles of crowdsourcing.^15,16,24 Based on our review, we defined crowdsourcing as the process of inviting a group of individuals to complete a task, attempt to solve a problem, or develop innovative ideas. In ten cases, individuals from diverse backgrounds were invited without restriction (i.e., anyone with access to the invite was allowed to take part).^{6,7,9,12,17,18,20-23} In six studies, the crowd was more narrowly defined around a target population.^{14-16,19,25,26} They shared their original ideas to answer questions or propose solutions.

Categories of crowdsourcing

We identified four categories of crowdsourcing activities, including open contests, hackathons, open forums, and incident reporting systems. Open contests, also called innovation challenges or innovation contests, are a subset of crowdsourcing approaches that solicit creative contributions from the target populations, evaluate submissions, celebrate finalists, and share the finalist ideas. Open contests were the most common crowdsourcing model used for HIV and sexual health research. Open contests have been used to invite target communities’ contribution to design ideas, images and videos on how to promote condom use and HIV testing.^{6,9,18,20-23,27} Hackathons are intensive, approximately 72-hour contests that bring people together to complete a specific task. For example, a hackathon could help develop HIV test promotion campaigns.^18,19 For both open contests and hackathons, entries were evaluated according to pre-selected criteria and then several finalists were selected to receive prizes. Open forum refers to Wiki-like platforms for individuals to contribute to a topic of interest. For example, the open source medical record system, OpenMRS, invites implementers and developers to communicate problems and solutions through the system.²⁴ Incident reporting system refers to an established online system that allow survivors or witnesses to report relevant incidents to a shared platform that is accessible by the public. Our review found past incident report and immediate report systems for sexual abuse or sexual harassment experiences. The former allowed adult victims who were abused in their childhood to voluntarily and describe what had happened to them and express their opinions or suggestions.^15,16 The latter was an immediate incident reporting system that allowed sexual harassment victims or witnesses to submit an immediate report to a platform via SMS text messages or social media tools about their experiences which then plots the locations where the sexual harassment incident occurred.¹⁷

Crowdsourcing phases

We identified four common phases for crowdsourcing approaches: 1) preparation, 2) problem framing and crowd solicitation, 3) judging submissions, and 4) sharing selected submission(s).

Phase 1: Preparation

The first phase of many crowdsourcing approaches was preparation.^{6,9,15,16,19,20,22,24} Activities included forming steering committees and/or advisory boards that establishes the rules and identifies the problem(s) to be solved,^{6,15-20,23,25} with some groups also conducting qualitative research to support the design of the crowdsourcing activity.¹⁴ All steering committees involved experts and people who have lived experience of the disease. The crowdsourcing activities were led by governmental authorities or other decision-makers,^15,16 research institutes,^{6,7,9,14,17-22,24} or non-governmental organizations.²⁵ They initiated, organized, implemented and monitored the whole activity. Some research groups noted the importance of including individuals from key populations in the steering committee, especially those who are marginalized and vulnerable, because the absence of their voice may reduce the effectiveness and trust in the activity.^20,25 Regular steering committee meetings was a common approach to follow up on progress and make refinements.²⁰ Workshops, discussions or other training activities were important strategies to strengthen the relevant parties’ capacity and leadership.²⁴ One study assessed the feasibility before launching a crowdsourcing contest using focus group interviews with key populations.¹⁴ Through the feasibility evaluation process, they developed logos, contest names, strategies to engage the community, and other contest promotion materials.¹⁴

Phase 2: Problem framing and crowd solicitation

The second phase includes framing the problem and promotion of the activity to the crowd. Online promotion strategies included Facebook, Twitter, Instagram, emails, open Skype calls, Weibo (a popular microblog used in China), WeChat (a Chinese instant messaging mobile application), and official websites.^{6,7,9,17-19,21,23,25} In-person promotional events, such as class lectures, community-based open forums, recreational activities, special events such as World AIDS Day, were adopted by other studies.^9,19,25 Local radio, which reaches a large number of people in LMICs, was also used to advertise the activity.⁹ The crowdsourcing approach led by UNAIDS used individual network distribution, encouraging young people connected to sexual health activism to spread the activity information to family, friends and acquaintances.²⁵ In-person engagement events were considered key for soliciting submissions from those who had limited access to the internet.^9,23,25 One study received more contest submissions from people who attended in-person events compared to those who did not attend in-person events.⁹

Some studies had strictly defined formats for submission. Six studies solicited images, videos, audio files, and texts with pre-determined size limits,^{7,9,14,18,19,23} and one study allowed victims or witnesses to immediately report the time, location and case description to a sexual harassment incident reporting system.¹⁷ Two studies had relatively flexible formats for submission without limiting the size of the submission to allow submitters to freely express what matter to them.^16,25

We identified various forms of incentives for participation. For open contests and hackathons, material prizes (e.g., iPad, cash, or other gifts) and coverage of participation expenses were awarded.^9,19,21,23 Non-financial benefits such as feeling empowered due to participation, having their voice heard, and contributing to the fight against HIV were also considered important motivators for participation in the crowdsourcing activity.^6,16,17,25

Phase 3: Evaluating submissions

Submissions were judged by crowd and expert judges based on pre-determined criteria.^6,7,9,18-23 The judging process was mainly to ensure quality of submissions and to select top ones for future interventions, health education or health advocacy purposes.^7,18,19,23 All crowdsourcing open contests and hackathons had the judging phase and submissions scored according to the pre-determined evaluation criteria. Fairly selecting judges from local key groups was important to increase community ownership of the event and to incorporate local preferences.²⁰ Using pre-determined evaluation criteria helped improve judging consistency across different judges.²⁰ Other crowdsourcing models had other forms of evaluation of their crowdsourced data such as analyzing the content of submissions.^15-17,22,25

Phase 4: Sharing the solution(s)

All crowdsourcing activities shared their outputs in some way and the sharing was sometimes nested within the process of inviting the crowd to judge submissions. We defined the outputs of a crowdsourcing activity as outputs of any format, including generation of new knowledge (e.g., perceptions of HIV cure),²¹ creation of new products (e.g., images, videos, logos and reporting platforms)^{6,15-17,23,24} or intervention materials (e.g., public health intervention materials),^7,18,23 development of community engagement campaigns (i.e., engaging youth in AIDS response),²⁵ social media activism (e.g., undermining the climate of social acceptability for sexual harassers),¹⁷ policy changes (e.g., institutional priority setting and guidelines for handling sexual abuse and preventing it),^16,25 adoption of new legislations,¹⁵ further research and education programs,¹⁵ and change in medical practice.¹⁵

Some open contests and hackathons shared submissions during the evaluation process and after finalists were selected.^{9,19,20,22,23} Presenting results to governmental authorities or other decision-makers facilitated policy change, and releasing results on websites and press conferences to inform sexual abuse victims about the ongoing process built up people’s trust and readiness to use the incident report system.¹⁵ Incident report data were used to convince shopkeepers, police officers and restaurant owners to begin intervening once they see sexual harassment in their neighborhoods. When they agreed, a “harassment free zone” sticker was given to them, illustrating to customers that their business was a harassment free zone.¹⁷

Crowdsourcing activity evaluation

Most crowdsourcing activities were evaluated in some way. The first form was to evaluate the crowdsourcing activity itself. Assessments included quantitative data about traffic (the number of webpage visits, followers, and reach of unique users)⁹ and qualitative evaluation of the content of submissions (e.g., emerging themes from the text submissions).^{6,9,15,16,21,25} The second form of evaluation focused on examining the level of community engagement as a result of the crowdsourcing activity. One study evaluated community engagement and the impact of participation in an open contest to promote sexual health using qualitative interviews.²² They found different forms of engagement activities, including submitting an entry to the contest and helping organize the contest events.²² Participating in a contest helped increase knowledge in the topic, multi-sectoral collaboration, and re-orientation of professionals toward people-centered sexual health programs.²² The third form of evaluation was through randomized control trials to evaluate the effectiveness of crowdsourced intervention materials, compared to conventional sexual health campaigns. For example, two studies from China evaluated the effects of crowdsourced HIV testing promotion approaches in MSM and concluded that it can significantly improve HIV test uptake.^7,18

Strengths and weaknesses

Crowdsourcing in health has strengths and weaknesses (Table 2). There may be higher potential for innovation due to soliciting crowd wisdom, encouraging collaboration from multiple sectors, empowering key/marginalized populations by integrating their perspectives into high-level policy making process, and creating intervention strategies that are cost-effective and locally relevant with strong potential for scalability. Meanwhile, disadvantages include some crowdsourcing projects only relying on internet channels which limits participation. Some open contests, though, included a series of in-person events to mitigate such an effect.^9,21-23,25 Formulating a call for entries that is engaging and widely disseminated can be challenging. For open contests and hackathons, these approaches are relatively short-term, and have potential risks of exclusion of community members from steering committees, biased crowd judging, exploitation and insufficient recognition of contributors, and limited sharing.²⁰

Table 2.

The strengths and weaknesses of crowdsourcing approaches

		Articles
Strengths	Higher potential for innovation due to heterogeneity of knowledge in the crowd compared with a panel of experts	1
	Encourages collaboration between different groups, fields, and sectors	1-5
	Empowers key populations	1, 3, 6-9
	Builds community capacity	5, 10
	Integrates grassroots perspectives into high-level strategy and policy process	6, 11, 12
	Minimal cost compared to social marketing for public health intervention development	5, 7-9
	Creates messages/strategies that are locally relevant and feasible to implement	5, 7-9
	Documents events of interests; mitigates the fear of stigma and retaliation through anonymous reporting	10-12
	Strong scalability and wide coverage of key populations	5, 10
Weaknesses	Over-reliance on internet channels and ignores individuals who lack internet access	1, 4, 13
	Risk of too few submissions if the topics are non-sensitive to populations of interest	1
	Open contests and hackathons are temporally transient and relatively short term	1, 4, 13
	Open contests have potential risks of excluding community members from steering committee, biased crowd judging	1, 4, 13
	Open contests and hackathons are subject to exploitation and insufficient recognition of contributors, and limited sharing	1, 4, 13
	Incident report systems focuses on collecting survivors’ experiences of sexual abuse/harassment but not directly preventing it from happening	10-12

Crowdsourcing for implementation science

Based on the RE-AIM framework, ²⁸ we analyzed how crowdsourcing may contribute to implementation science. RE-AIM is a model commonly used to evaluate the impact of public health interventions in five dimensions: reach, efficacy, adoption, implementation and maintenance.²⁸ Crowdsourcing may be useful for several parts of implementation research (Table 3). Crowdsourcing may increase the reach of interventions by engaging large groups of local at-risk individuals in the development and evaluation of the intervention itself.²⁵ Crowdsourcing may increase adoption of an intervention by including key policy-makers, implementers, and key populations in decision making roles.^15,16 Crowdsourcing programs could be easier to implement because it draws on local styles, preferences, and culture which are known to influence implementation.¹⁸ Regarding maintenance, crowdsourcing often enhances trust in health services that is important for maintaining an intervention over time, and can create strong community ownership, resulting in more active engagement in sustaining or scaling up programs.²⁶

Table 3:

Crowdsourcing for HIV and sexual health implementation science based on the RE-AIM framework¹⁴

	Potential role for crowdsourcing	HIV examples and references
Reach	1. By using local and audience-friendly language suited for specific contexts, crowdsourcing can reach individuals from communities who may have language barriers.	CrowdOutAIDS led by UNAIDS organized open forums in all languages spoken in the BRICS countries (Brazil, Russia, India, China and South Africa) to minimize language difficulties. 6
	2. Crowdsourcing engages at-risk targets groups during intervention development and evaluation, increasing the potential reach of the intervention.	1. CrowdOutAIDS reached 3,295 youth participants from different regions worldwide online and 1,605 participants offline.6 2. Over 14 weeks, the in-person community events of an HIV cure open contest reached 144 participants in North Carolina. The online promotion channels reached 80,624 unique users.4 3. A crowdsourcing intervention study to promote HIV test among Chinese MSM showed that their messages reached 91.4% of men recruited in the study, and 67.1% of them shared at least one image or text.15
Efficacy	HIV and sexual health interventions developed using crowdsourcing methods have increased HIV test uptake rates among MSM.	1. One RCT showed that, compared to the control period, the crowdsourcing intervention significantly improved HIV test uptake rate among Chinese gay men, especially HIV self-testing. 16 2. Another RCT comparing the effects of crowdsourced HIV test promotion videos and expert driven health marketing videos found that the former had similar effects on improving HIV testing but higher community engagement.8
Adoption	Crowdsourcing may increase adoption of an intervention by including key stakeholders, including key populations, in decision making roles.	1. The child sexual abuse incident reporting program involved both church organizations and government in the program development process. This activity resulted in more actions against child sexual abuse by both churches (i.e., adopting guidelines and framework for preventing and managing child sexual abuse, as well as research project on the topic) and the government (i.e., amending laws to strengthen victims’ rights).12, 17
Implementation	1. At the individual level, crowdsourcing may be easier to implement because it draws on local user preferences, styles, and culture that are known to influence implementation.	1. A RCT evaluating the effectiveness of a crowdsourcing intervention on HIV test uptake among an MSM sample showed that 62% of them tested for HIV during the study period.16
	2. From an implementer perspective, an intervention designed for the local health system may be easier to implement.	2. Following the guidance from the study team, local CDC and gay community organizations in all eight study sites in an intervention program implemented a story contest with contextual adaptations to promote HIV testing among local MSM,16 which resulted in increased likelihood of facility-based HIV testing behaviors.15
Maintenance	1. From the individual perspective, crowdsourcing may increase trust in health services that is known to be important for maintaining an intervention over time.	1. In a crowdsourcing intervention study, among MSM participants, the proportion who maintained HIV testing every three months after the intervention ranged from 32.0% to 39.4% at the end of the 12-month study, compared to 19.1% to 29.0% before the intervention.16
	2. Crowdsourcing activities also have strong community ownership, resulting in more active engagement in maintaining or scaling a program.	2. The open source electronic medical record application (OpenMRS) was established and applied to HIV/AIDS and Tuberculosis patients and treatment information management in developing countries. This system enabled local implementers and developers to freely collaborate and discuss technical issues pertaining to the development of OpenMRS system. This model has been maintained since its first establishment in South Africa in 2006 and replicated in several other African sites.18

R: Reach, ie proportion of the target population that participated

E: Efficacy, ie success rate at changing desired outcomes

A: Adoption, ie proportion of target settings involved

I: Implementation, ie extent to which the program was delivered as intended. Measures of implementation has both individual and program levels. Measures of individual-level implementation include participant follow-through or “adherence” to regimens, while measures of program-level implementation mainly include to what extent staff members deliver the intervention as intended.

M: Maintenance, ie extent to which the program outcomes (at the individual level) and program availability (at the setting level) were sustained over time

DISCUSSION

We conducted a scoping review and qualitatively examined studies utilizing crowdsourcing in HIV and sexual health research to identify ways in which previous studies applied crowdsourcing methods, and identify strengths and limitations of such approaches. Given that crowdsourcing methods are used increasingly in HIV and sexual health research, it may be worthwhile to create a common framework for ensuring quality in designing and reporting future crowdsourcing studies. A practical guide for conducting open contests was published by World Health Organization Special Programme on Training and Research in Tropical Diseases.²⁷ The emergence of common phases in crowdsourcing activities in our review further supports the potential for creating such a framework.

Researchers used a variety of crowdsourcing approaches to address HIV and sexual health challenges across different geographical and cultural contexts. Open contests, in conjunction with hackathons, were the most commonly used crowdsourcing model to develop cost-effective public health intervention programs. This approach is particularly appealing in HIV research as it enables large groups of people who are often marginalized or difficult to reach, to create culturally appropriate solutions to problems facing their communities. Crowdsourcing is thus a practical means for effective community engagement,²⁹ which is recognized by the World Health Organization as a key component to the HIV response.¹ Though less commonly used, incident reporting system was another way to harness the wisdom of the crowds in a timely manner, for instance, the real-time reporting platform to document sexual harassment incidents.¹⁷ A similar approach has also been implemented to protect sex workers from dangerous clients, demonstrating the capacity for scaling this up in other nations.³⁰

Our review has demonstrated several strengths of using crowdsourcing methods in HIV and sexual health. Multi-disciplinary collaboration and heterogeneity are key features which maximize the potential for innovations by aggregating crowd wisdom. Another advantage is that crowdsourcing activities provide an anonymous channel for people to make their voice heard without fearing stigma. This illustrates the potential of crowdsourced approaches in empowering marginalized or vulnerable communities and integrating their perspectives into priority setting as well as facilitating people-centered policy making to tackle stigmatized sexual health problems.

Our scoping review also identified potential weaknesses of crowdsourcing that can be improved. One major concern associated with crowdsourcing activities is that they often rely on having internet access and thus constrain participation;²⁵ however, in-person engagement events have been used to mitigate this risk.^14,25,31 A combination of online and in-person promotional strategies may help to reach those who might be difficult to approach in-person as well as to reach those who have difficulty accessing the internet.^20,23 In addition, open contests are relatively short term, and it is challenging to sustain community engagement. Nevertheless, a series of sequential short-term contests may help sustain interest and engagement.³² Open forums for developers and implementers of an open source medical record system to communicate and the immediate incident reporting system that allows sexual harassment victims or witnesses to submit an immediate report are good crowdsourcing examples that can sustain engagement. The former started in 2006 while the latter started in 2010. Both are ongoing programs and have been expanded to a wider international community.^33,34

Our review has implications for implementation science and the standardization of crowdsourcing methods. Our data suggest that crowdsourcing may be useful for several components of implementation science. Our synthesized data suggests that there are some cross-cutting themes in crowdsourcing research studies. The common phases that we identified in this review may serve as the basis for subsequent standardization of crowdsourcing methods. The WHO/TDR Practical Guide on Crowdsourcing in Health and Health Research provides another foundation, but more formal guidance is needed.

CONCLUSION

Crowdsourcing methods are increasingly used in HIV and sexual health research. They include a diverse set of activities but share common phases with one another. These common elements may be useful for producing a common framework for designing and reporting future crowdsourcing projects. Further research related to crowdsourcing in the context of implementation science is needed.