How to (or how not to) implement crowdsourcing for the development of health interventions: lessons learned from four African countries

Abstract

Crowdsourcing strategies are useful in the development of public health interventions. Crowdsourcing engages end users in a co-creation process through challenge contests, designathons or online collaborations. Drawing on our experience of crowdsourcing in four African countries, we provide guidance on designing crowdsourcing strategies across seven steps: deciding on the type of crowdsourcing strategy, convening a steering committee, developing the content of the call for ideas, promotion, evaluation, recognizing finalists and sharing back ideas or implementing the solutions.

Key messages.

• Globally, crowdsourcing strategies are increasingly being used to identify innovative and community-centred solutions for diverse health problems. The use of crowdsourcing has shown great potential for engaging communities and co-developing solutions that could improve health outcomes.

• It is important to tailor crowdsourcing strategies to specific aims, intended audiences and context, including a consideration of access to technology. It is helpful to remain flexible and iterative in the implementation of crowdsourcing strategies and to remember that crowdsourcing is often a means to an end rather than a panacea. Hence, risks should be carefully considered, and mitigation strategies should be put in place for the best outcomes.

Introduction

Crowdsourcing strategies are increasingly being used to identify innovative and community-centred solutions for diverse health problems. Crowdsourcing is defined as having a large group solve a problem and then share their solutions with the public (World Health Organization, UNICEF, 2018). Crowdsourcing typically employs a ground-up approach, including beneficiaries in the design and implementation of solutions. Through co-creation with end users, crowdsourced interventions may have greater potential for impact and sustainability (Mathews et al., 2017; Wang et al., 2020).

There are three common types of crowdsourcing strategies: contests, designathons and online collaborations. Contests are open calls for ideas, with solicited ideas evaluated and exceptional entries awarded a prize. Designathons involve a call for ideas followed by collaborative workshops to refine ideas and/or solve the problem statement, with prizes awarded to top-ranked solutions. Online collaboration involves the use of online platforms where individuals can discuss and share ideas openly but usually does not include a competition with prizes.

In public health, the use of crowdsourcing has accelerated over the last decade in diverse settings, as evidenced by the growing number of publications on the topic (Ranard et al. 2014; The National Academies of Sciences Engineering and Medicine, 2011; Wang et al., 2016; Wazny, 2018). A number of these publications describe the use of crowdsourcing to support intervention design to improve public health outcomes, including the development of interventions to improve the uptake of Coronavirus disease 2019 booster vaccines, a peer-led intervention to improve safer use of a dating app and interventions to increase youth HIV testing (Iwelunmor et al., 2020; Wong et al., 2020; Kpokiri et al., 2021; Böhm et al., 2022). Despite widespread applicability and increased use, crowdsourcing is not a panacea for the co-development of public health interventions. Some risks exist for which mitigation strategies should be in place when developing a crowdsourcing strategy. The common risks include failure to organize and engage the steering committee, inability to engage the community and poor judging of the entries received (World Health Organization, UNICEF, 2018). Others include the variations in crowdsourced outcomes with conventional guidance and practices. Meaningful engagement of populations, particularly vulnerable populations including youth, affected by the public health outcome of interest may require additional considerations and resources. In addition to existing frameworks (World Health Organization, UNICEF, 2018), there is a need for specific guidance on ‘how to’ and ‘how not to’ implement crowdsourcing strategies.

In this paper, we highlight lessons learned through implementation of challenge contests and designathons in four African countries: Nigeria, Senegal, Zambia and Zimbabwe, to complement the guide developed by the Special Programme for Research and Training in Tropical Diseases (TDR) (World Health Organization, UNICEF, 2018). We consider our lessons applicable to other African settings and to crowdsourcing strategies that aim to co-develop interventions. Four of the five crowdsourcing strategies focused on involving youth in intervention development; our lessons learned are focused heavily, but not exclusively, on campaigns to engage youth.

Steps for implementing crowdsourcing

We describe seven steps in implementing a crowdsourcing strategy, from deciding whether to use crowdsourcing and designing the call for ideas to sharing and implementing the final crowdsourced ideas. For each step, we provide a short description of what is involved, the lessons learned and how to optimize each step. We draw on lessons learned from a campaign in Nigeria to source ideas on how to promote HIV self-testing among youth (4 youth by youth); campaigns in Senegal (Bayal Sa Waar) on how to improve sexual health communication between parents and adolescents as a means of improving access to relevant healthcare services; a campaign in Zambia (Yaba Guy Che ‘for the guys’) to identify strategies to improve men’s uptake of HIV testing services and two campaigns in Zimbabwe on using art to engage youth in health (The Art of Health) and how to promote general health check-ups among youth (Y-Check; Table 1).

Table 1.

Summary of key components of crowdsourcing strategies conducted in Nigeria, Senegal, Zambia and Zimbabwe

Crowdsourcing domain	Nigeria (4 youth by youth)	Senegal (Bayal Sa Waar)	Zambia (Yaba Guy Che)	Zimbabwe (Art of Health)	Zimbabwe (Y-Check—Zve Hutano)
Aim	To identify strategies and ideas on ways to promote HIV self-testing (HIVST) among young people in Nigeria.	To identify strategies to increase individuals’ (15–19 years of age) access to sexual and reproductive health services.	To identify strategies and ideas on ways to reach men with HIV testing services.	Use of art (e.g. music, drama, art, dance, photography and theatre) to engage youth on matters pertaining to their health and well-being.	To solicit ideas, images and strategies to encourage youth to seek regular check-ups. Participants developed the Y-Check brand through logo design and theme song creation.
Dates open	1 October to 16 November 2018	20 May to 26 June 2022	25 July to 30 September 2022	17 February to 17 April 2021	21 October to 19 November 2021
Design	Contest with designathon	Contest (urban) and designathon (rural)	Contest	Contest	Contest with designathon
Focus population for participation	Youth aged 10–24 years	Youth aged 15–21 years	Men and women aged ≥18 years	Youth aged 18–30 years	Youth aged 10–19 years
Population of end user/beneficiaries	Youth aged 10–24 years	Youth aged 15–19 years	Young men
How the contest was promoted	Online (Facebook and Instagram) and in person (secondary schools, universities, etc.).	Online (videos and posters on Instagram and WhatsApp) and in person (clubs and community) with fliers, posters and a van with a loudspeaker.	Online (videos and posters on Twitter, WhatsApp and Facebook) and in person (fliers and talks during walkabouts in the community at busy areas).	Social media (Facebook, Instagram and Twitter) and in person (youth mobilizers in the community, secondary schools, universities and community-based organizations that work with young people).	In person only; a mobilization team targeted youth ‘hot spot’ areas and entered local schools where they engaged with the learners and teachers. Community-based organizations working with young people were also leveraged.
The ways in which entries were received	Online (Google forms, WhatsApp or email) or offline (paper-based version).	Online (WhatsApp) and in person (via a collection box).	Online (WhatsApp and Google Docs) and in person via collection boxes at churches and health facilities.	Online (Google Docs, email or WhatsApp) and in person.	Online via WhatsApp or physical entries through the teachers or the community mobilization team.
Total entries received/judged	Nine hundred and three entries; 769 eligible and judged.	Six entries received (contest); four judged. Fourteen group entries (designation); ideas from five groups judged.	One hundred and forty-two entries received: 60 judged.	One hundred and fifty-three entries received; 60 judged (10 shortlisted in each category).	One hundred and sixty-five entries received: 156 judged (10 shortlisted in each of art and music categories).
Key lessons learned	Consider multiple platforms for receiving submissions. In resource-constraint settings, allow offline promotion and participation to include those with limited online presence and internet access.	Designathon more appropriate design with adolescents; ensure judges are clear on the scope and are not too critical during any public assessment.	Clarify that only one idea should be included per submission; allow a range of ways to submit ideas to accommodate varying literacy/phone ownership levels. Where feasible, do not concentrate to one geographical area.	Consider multiple platforms for information dissemination, remunerate youth for their effort and develop capacity of youth where possible.	In low-resource settings, it is key to provide a larger emphasis on offline submission mechanisms, especially when dealing with young adolescents who have limited access to phones or the internet.

Step 1: deciding the type of crowdsourcing strategy to be implemented

Crowdsourcing strategies can be used across diverse settings, as demonstrated by our five examples. Crowdsourcing may be particularly useful in settings where there is a potential to mobilize diverse groups of people and where the topic is of interest to the public (beyond academic research), and a crowdsourcing strategy (a contest, designathon or online collaboration) can be implemented (Han et al., 2021). Deciding on the type of crowdsourcing strategy to be used requires consideration of the population and their access to technology and the public health topic of interest.

In settings where and with populations for whom access to the internet and/or phones is limited and with populations who have lower literacy, contests and designathons are preferable to online collaboration. Contests, which typically require 4–6 months for organization, can be run in-person, online or hybrid. Contests require communication expertise to encourage submissions and reach intended populations (see Step 3) and have been used to develop HIV (self)-testing interventions with youth and men (Table 1) and to increase youth access to health services, reach underserved groups and encourage creative community engagement (Iwelunmor et al., 2020).

Designathons are organized in three stages, including preparation, an intensive period of collaborative work (typically 48–72 h) and follow-up activities. Designathons can be organized in-person, online or hybrid. They have been used to finalize public health interventions (Tahlil et al., 2021) and develop consensus documents (Kpokiri et al., 2022). Designathons may require more resources than contests because they often require food, accommodation and other logistical support. However, designathons may be preferable to contests when involving youth as more intense interaction may result in more appropriate and feasible solutions. In Senegal, the designathon, which provided one-to-one interaction with youth >2 days, led to more relevant ideas when compared with a related contest (Table 1).

Finally, online collaboration provides another approach to crowdsourcing but requires an internet-enabled device. For example, the UNICEF U-reporter uses unstructured supplemental service data to solicit information from youth in low- and middle-income countries. This approach may be preferred to engage populations who are stigmatized and/or solicit ideas on stigmatized health topics as they can remain fully online, and therefore less visible, and can leverage existing social network platforms.

Step 2: deciding on the composition of and engagement with a steering committee

Steering committees are an important feature of crowdsourcing, providing valuable input into the design, promotion of the call and how to receive entries. The steering committee should be diverse, including individuals from the focus population, public health experts and practitioners/policymakers, among others, but should have sufficient interest and expertise in the topic to contribute meaningfully. The lead researcher or a designated member of the study team chairs the steering committee meetings, but the deliberations and final course of action lie with the steering committee members.

Typically, participation on the steering committee is voluntary. Steering committees have optimally engaged this way; hence, inviting members with interest and passion for the topic or focus of study is the key as this will be a big part of their motivation in addition to other crowdsourced outputs. As such, to support continued engagement, the frequency of meetings should be minimal, e.g. a monthly 60-min tele-conference, and communicated clearly in advance of individual commitment to the committee. The roles of the steering committee members should be made clear and should be agreed by all members but kept open for any further roles and responsibilities. We recommend that the committee be engaged in all aspects of the crowdsourcing strategy, leveraging their expertise and networks to increase engagement and participation and to promote the call for ideas. Members can also serve as judges.

In Zambia, the steering committee comprised academics from national universities, researchers, community members and Ministry of Health staff (central and district level). The committee had one physical meeting, and subsequent communication and meetings were online. Members provided feedback and guidance when developing the call, ranging from wording used to the visual characteristics of the characters used (Figure 1).

Figure 1.

Posters used (from left to right) in Nigeria, Senegal, Zambia and Zimbabwe to crowdsource ideas from the public

Step 3: deciding the content of the call for ideas

Encouraging individuals to submit their ideas requires images that capture the attention of the intended audience and text that is clear and concise. In designing the call, we recommend having communication experts on the team. For marketing purposes, images to promote the call could also be co-created with the intended audience (Finley et al., 2020). In Zambia and Zimbabwe, the study team engaged a local marketing company to develop promotional images and refined these with the steering committee. In Zambia, the content of the posters included three different characters (Figure 1) to appeal to a broader population, including women.

There is a need to consider the specificity of the call. A call that is too broad may not be clear, but the one that is too narrow might not resonate with many individuals. We found that iteratively co-developing the call, including the images and the text, in partnership with community members and the steering committee can ensure that balance is achieved.

The eligibility criteria need to be clear to avoid crowdsourced ideas that are too varied to draw any resulting solutions. For calls relying solely on posters, brevity is particularly important, whereas longer text can be facilitated through videos and audios. In Zambia, the strategy included posters and an animated video. It was important to ensure that messaging across the two platforms was similar, but the video allowed for elaboration on the purpose of the call and formed a useful complement to the posters.

Details required in the call include deadlines, eligibility and judging criteria, prizes for finalists, timelines for when to expect feedback, if any, contact details for questions and details on how to submit one’s idea. It is advisable to provide different submission options to maximize reach; for example, in Senegal and Zambia, individuals could submit their ideas via WhatsApp or in person. Clear details on what can be submitted should be provided, alongside the demographic information that should be submitted. Where submission is online only, a clear and concise submission form should be included on the call website. This should provide a consent form, provide the ability to collect demographic data and allow a file upload of the submission. Information on the intellectual property rights for the ideas or products generated from the crowdsourcing should also be clearly stated. A core list of frequently asked questions can be helpful to be included at the end of the site.

Step 4: promoting the call for ideas

It can also be very useful to co-develop the promotion strategy with a community advisory board (CAB). In Senegal, the team formed two adolescent CABs, one in each site. The CABs were critical in the development of the protocol for promoting the call, including engaging local influencers, leaders and celebrities to promote the call on social media. The CAB members also disseminated the call on their own social media platforms and through their social networks.

The first component of the promotion strategy is defining the audience of the call. This may be different from the beneficiary population; for example, in Senegal, whilst ideas being sourced were to develop an intervention for individuals aged 15–19 years, the call sourced ideas from people aged 15–21 years to capture ideas from those who likely had recent experience in accessing sexual and reproductive health services. Similarly, in Zambia, although the call sourced ideas for interventions to reach young men with HIV testing services, the call was open to anyone aged 18 years or older. Where budget allows, it may be beneficial to hire a branding and marketing agency to reach a broader population. In Zimbabwe, the Art of Health engaged an agency to assist in branding and received more than 1050 submissions. Also important to consider is reaching marginalized groups, such as individuals with lower literacy, remote and older populations or disabled people. For these, enabling peer-based support and co-creation sessions may help to ensure access. In Zambia and Zimbabwe, mobilizers engaged with men and women, and youth face-to-face, respectively, to introduce the contest and encourage the submission of ideas. In Zambia, this allowed for ‘on-the-spot’ submissions from individuals without internet-enabled phones.

The second consideration is the type of media to promote the call. We advise a mix of online and in-person promotion; however, the most appropriate approach will depend on the intended audience. In Zambia, the promotion was via Twitter, WhatsApp and Facebook, complemented by regular in-person promotion in busy locations (e.g. markets and bus stations). Use of local radio stations, magazines or dailies should also be considered. Promotion should be continuous, with videos and images shared weekly until the deadline to maximize visibility. Decisions need to be made on ways to maximize exposure to the call. For in-person promotions, options can include print, audio or video or a combination of all three. The important consideration when selecting the dissemination channel is maximizing exposure. For calls focused on youth, it might be useful to engage schools, after-school clubs and community organizations, such as youth and sports clubs. The Nigerian 4 youth by youth call engaged youth leaders for in-person promotion in schools, similar to the strategy in Senegal, and used social media platforms with high youth presence such as Facebook and Instagram, as in Zambia (Iwelunmor et al., 2020).

Step 5: assessment of entries

After the deadline, entries can be assessed in one or more stages. A digital or paper scoring sheet can be used with points allocated for pre-specified judging criteria agreed by the steering committee (e.g. originality, feasibility and innovation). If there are few entries (e.g. <50), a one-stage process can be used with entries scored and finalists selected. Where there are many (e.g. >50) entries, a two-stage process can be helpful. In the first stage, a pre-determined minimum score can be used to short-list eligible entries, and in the second stage, those that met this initial minimum score are re-evaluated by different individuals from those who judged stage one. This two-stage process was used in Zambia, where close to 150 entries were received.

To increase rigour and fairness, it is advisable to have at least two or more independent judges score entries and for judges to declare any potential conflicts of interest. Additional judges can be available to provide a third or fourth opinion if the standard deviation between the scores is high or in cases tiebreakers are needed. Selected judges will depend on the contest but often include a combination of members of the study team, the steering committee, public representatives and others from minority or under-represented groups. Including steering committee members as the judging panel allows for transparency and continuity and may encourage greater commitment to the committee. Judges should have interest in the topic of the contest, be briefed on their role and be provided with a judging rubric and scoring sheets. Judges are encouraged to provide brief comments to justify score allocations. In-person judging, by way of open votes in an event, can also be employed for transparency. In that case, judges need to be sensitized on ways to provide feedback, particularly in the case of vulnerable populations. For example, the Nigerian contest invited finalists to pitch their ideas at an annual World AIDS Day event, and three top finalists were selected based on cumulative votes. Judges were required to provide feedback, to declare any conflicts of interest on submissions assigned to them and to recuse themselves from judging any submission where there may be a conflict of interest. Blind evaluation is advised by anonymizing the entries prior to judging.

Step 6: recognizing finalists

A critical step in crowdsourcing contests is recognizing the finalists. The manner by which finalists will be rewarded for their time and effort should be decided from the very beginning and clearly communicated. Input from the steering committee and community members is necessary, including CABs. The audience should be informed of the criteria to be used to determine finalists and of the prize for the winning ideas. All the crowdsourcing strategies implemented by the authors provided money as the prize, which was preferred by each focus population as a reward that would motivate participation.

After finalists are identified, announcements should be publicized and ideas should be shared with the community. In Zambia, a prize-giving ceremony was held with the top 15 finalists. At this ceremony, the top three finalists were revealed, and the ceremony was broadcast live on Facebook. In Senegal, designathon finalists were invited to present and share their ideas at a public event, followed by a ceremony where prizes were awarded.

Public recognition should be done with participant consent, taking care not to disclose any confidential participant information, e.g. regarding whether ideas are sourced from people living with HIV or other stigmatized populations. In addition to the prize for the top finalists, other prizes for exceptional ideas should be considered, such as mobile talk time or certificates for participation. This can be important to demonstrate the value of time and effort on the part of all who participated.

Step 7: sharing back, implement or pilot ideas

A common assumption is that recognition of finalists is the last step of a crowdsourcing strategy. However, a critical step is sharing the ideas with a broader audience and implementing the solution. Recognizing finalists and sharing the winning ideas can be done in tandem. In Zambia, alongside live streaming, various community members and organizations were invited to the prize-giving ceremony. In addition, a series of iterative workshops were held to refine the contents of a multicomponent intervention, the development of which was contributed by crowdsourcing; through these workshops, the winning crowdsourced ideas were shared with community-based organizations and policymakers.

Beyond sharing the winning ideas, pilot implementation of the crowdsourced ideas is recommended. First, this shows the audience crowdsourcing led to action, and secondly, crowdsourced interventions are likely to have greater ownership and, therefore, impact. The Nigerian call informed an ongoing cluster randomized trial that has influenced national policy on HIV self-testing (Iwelunmor et al., 2020). In Zambia, crowdsourcing formed a part of research study to co-develop a multicomponent intervention, and the study team submitted a funding proposal to evaluate the implementation of the strategy within the community where crowdsourcing was conducted.

Conclusions

This paper emphasizes the importance of tailoring crowdsourcing strategies to specific aims, intended audiences and context, including a consideration of access to technology. It highlights the lessons learned through implementing crowdsourcing strategies in four African countries to complement and build on the crowdsourcing practical guide developed by the WHO TDR (World Health Organization, UNICEF, 2018). The use of crowdsourcing has shown great potential for engaging communities and co-developing solutions that could improve health outcomes (Day et al., 2020). Where crowdsourcing strategies involve youth or other specific groups, their involvement in the steering committee, call for ideas and judging process is crucial as promotions may only reach sub-populations of the potential audience. Multiple judges should be engaged to review entries, and judges need guidance notes (rubrics in allocating scores to ensure that relevant contributions are not ruled out). Public recognition of finalists should be carefully planned to appeal to the intended audience and protect potentially stigmatized individuals or groups.

It is important to remain flexible and iterative in the implementation of crowdsourcing strategies and to remember that crowdsourcing is often a means to an end rather than a panacea. Potential risks should be considered with mitigation plans put in place. Despite limitations and risks, by implementing these key lessons learned, future crowdsourcing strategies can be strengthened.

Supplementary data

Supplementary data is available at Heapol Journal online.

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author.

Funding

The Art of Health was funded by the Wellcome Trust in 2021 through a Research Enrichment Public Engagement Grant [206316/Z/17/A/A]. Yaba Guy Che (‘for the guys’) was funded by the Medical Research Council through the Public Health Intervention Development Grant [MR/V031171/1]. The Y-Check Zve Hutano (‘our health’) contest was funded by the Medical Research Council [grant number MR/T043156/1]. The Nigerian 4 youth by youth contest was funded by the Eunice Kennedy Shriver National Institute of Child Health and Human Development [grant numbers UG3HD096929, UH3HD096929 and NIAID K24AI143471].

Author contributions

B.H. conceived the study design and concept. B.H., E.E.K., M.M.P., M.M.-A., M.T., C.D.C., F.N., A.M.D. and J.D.T. were involved in data collection and initial draft of the manuscript. B.H. and E.E.K. critically reviewed the article, and all authors read and approved the final version of the manuscript.

Reflexivity statement

This article carefully considered inclusivity and equity within the author groups. The authors have been included from both high- and middle-income countries. In terms of position, gender and ethnicity, the author group presents a good diversity within these dimensions.

Ethical approval.

No ethical approval was required for this study.

Conflict of interest:

None declared.