Codesigning COVID-19 booster promotion materials in online workshops with long-term care staff: a process evaluation

Abstract

When engaging communities in vaccine promotion efforts, it is critical to understand who has availability and interest in participating, and how participants experience the process. Guided by the Reach, Effectiveness, Adoption, Implementation, Maintenance framework, we evaluated an online codesign process with long-term care center staff to design coronavirus disease (COVID-19) booster promotion materials for their colleagues. Twenty-six staff joined codesign teams organized by self-identified race to culturally tailor materials during a cluster-randomized controlled trial with 40 centers in two US states. Data sources included surveys; interviews; screening, enrollment, and attendance data; fidelity assessment; codesigned materials; and focus groups. We report summary statistics and thematic analyses. We found that time constraints could impede enrollment. Most codesigners enjoyed collaborating and identified their individual contributions. Many linked having shared characteristics with their teammates (e.g. race, gender, age) to a feeling of connection. A few reported feeling motivated to engage in booster promotion. The process resulted in messages aligned with known psychological antecedents of vaccination. Considering recruitment challenges and limited participation of direct care staff, organizers should consider shorter codesign processes. Organizing teams based on shared characteristics could promote comfort. Community input should inform codesign structure and team composition to achieve optimum enrollment and engagement.

Introduction

During the coronavirus disease (COVID-19) pandemic, vaccinating long-term care workers was a US public health priority to protect their health and prevent disease transmission because a majority of residents have a high risk of severe COVID-19 illness and hospitalization due to older age and underlying chronic conditions [1, 2]. The original vaccine series was federally mandated for US skilled nursing facility (SNF) staff [3] but the booster vaccination was not. According to the Centers for Disease Control and Prevention, as of March 2023, 22.8% of the national nursing home workforce was up-to-date with COVID-19 vaccinations—defined as completing a primary series within the preceding 2 months or receiving the bivalent booster [4]. This represents a steep decline from the previous year, in which 80% of long-term and home health care staff had received the primary series; of those, 61% had received a booster [5].

Increasing vaccination coverage is important because many staff at US SNFs and assisted living communities (hereafter described collectively as long-term care centers, LTCCs) could be at risk for poor COVID-19 outcomes, which are linked to older age [2] and low income [6, 7]. In addition, people of colour are disproportionately represented in direct care roles, with 28% of people in these roles identifying as Black [7]. During the pandemic, Black communities experienced higher rates of poor COVID-19 outcomes, [6, 8, 9] linked to factors such as structural racism [10–12], residential segregation [10], and poor primary care access [13]. Although racial and ethnic disparities in receiving the primary vaccine series in the USA had narrowed by 2022, racial disparities in booster vaccination persisted. When compared with the White population, all other racial groups except Asian had lower uptake [14].

The recently updated model of the seven psychological antecedents of vaccination (7Cs) provides a framework that situates factors contributing to these disparities and low uptake in LTCCs, notably ‘Social Conformism’ and ‘Confidence’ (in the system and in the safety and efficacy of the vaccine) [15–17]. ‘Confidence’ has been undermined by misinformation and conspiracies about the vaccines and their development, which often target specific communities (e.g. by faith) on social media [18]. Research with Black communities identified concerns related to ‘Confidence in the System’ based on: (i) historical mistrust and mistreatment because of abusive, unethical research, (ii) current mistrust of the government and health care related to structural racism and continuing injustice, and (iii) fear of inequitable treatment or being an object of experimentation [19, 20]. One study carried out in SNFs surfaced concerns related to ‘Confidence’ and ‘Calculation’, which refers to deliberation about risks and benefits [15]. These included the rapid development process, side effects, infertility risk, safety during pregnancy, and beliefs that the vaccine or booster caused COVID-19 or was ineffective for new variants, with most staff reporting that social media and friends were their main information sources [21]. The varying reasons for vaccine hesitancy [22] could explain in part the lack of success of evidence-based, theory-driven interventions to increase vaccine uptake in SNFs [23] and health care [24]. In addition, high levels of hesitancy in the workplace likely contribute to low uptake considering that ‘Social Conformism’—defined as health care worker perceptions of acceptance of the COVID-19 vaccine in their workplace—is associated with intention to get the vaccine [16].

These circumstances suggested a need for community-engaged strategies to develop tailored, culturally relevant interventions for specific populations [25–27]. To create an intervention to increase booster uptake among LTCC staff, our study team chose a codesign approach because it was well suited to developing materials that would address varying perspectives about boosters. Codesign approaches such as boot camp translation (BCT) [28, 29] engage community members affected by health issues to collaborate on the design of contextually relevant solutions, programs, or materials. By engaging LTCC staff, codesign holds promise for addressing ‘Social Conformism’, i.e. skepticism about the vaccine in the workplace, and ‘Confidence’ because trusted staff could be well positioned to influence the opinions of their coworkers.

When conducting codesign in the context of mistrust and concerns, it is important to explore whose views are represented by evaluating who does and does not enroll in codesign and how participants perceive team dynamics and their individual contributions. This article describes our mixed-methods process evaluation of a 10-week codesign workshop with LTCC staff in teams organized based on self-identified race. This workshop was offered as part of a cluster-randomized trial to compare the effectiveness of codesigned materials versus standard government materials. Guided by the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, and Maintenance) [30], we evaluated four of the five domains: Adoption of the codesign process by LTCCs, Reach to LTCC staff included in codesign teams, Implementation of codesign activities, and Effectiveness of the process from participants’ perspective. The results of the trial testing whether the codesigned materials were more effective at increasing the booster rate and improving staff willingness to promote the booster will be reported elsewhere. The goal of this article is to provide lessons to inform codesign in LTCCs—particularly related to vaccine promotion—and codesign efforts more broadly.

Methods

Cluster-randomized trial context

For the trial, 40 residential LTCCs in Georgia and Washington state were recruited with support from partnering organizations and directly by the study team, which compiled a database of over 2000 LTCCs using public data sources. LTCCs were randomized to either codesigning and distributing COVID-19 booster promotion materials or distributing government booster promotion materials [31]. Staff from 20 LTCCs randomized to codesign were invited to join codesign teams. This study was approved by the prime site’s Institutional Review Board, and all participants gave informed consent.

Codesign context

Codesign participants (codesigners)

Codesign teams were intended to be based on self-identified race, ethnicity, and/or language, to create a sense of shared identity and develop tailored materials. To determine teams, we reviewed demographic data from a baseline LTCC staff survey at all sites and spoke with LTCC administrators about racial and ethnic groups represented among staff. We determined that the most feasible groups were Afro-Caribbean, Black or African American, or White teams in Georgia; and African immigrant, Asian or Pacific Islander, Black or African American, Latinx (gender-inclusive term), or White teams in Washington state. We intended each LTCC in the codesign arm to send ≥1 representative to each team in the corresponding state. We sought codesigners with a variety of roles, particularly low-wage jobs, considering that 65% of direct care staff in long-term care earn <$35 000 annually [7].

Recruitment included in-person outreach, posters in English and Spanish, recruitment emails distributed by LTCC administrators on behalf of the research team, and emails or text messages sent directly to staff who indicated in our baseline survey that we could contact them about study activities. These materials described the study objective, codesigner role, codesign process, time commitment, and incentive. To avoid any perception of coercion, administrators did not recruit individual staff members directly. Research assistants screened staff who completed the codesign interest survey and met inclusion criteria by phone. Primary inclusion criteria were: (a) willingness to promote the booster to coworkers, measured with a 10-point Likert scale survey item [31], (b) currently employed at an LTCC in the codesign arm, (c) not being an administrator, and (d) no concerns identified about willingness to collaborate.

Codesign curriculum source

We created, tested, and evaluated an online codesign process (Fig. 1) inspired by the BCT codesign method [28] and the Immunity Community intervention [32]. BCT has been used to create behaviour-change and preventive-care materials, including to promote human papilloma virus (HPV) vaccination [29, 33–35], and adapted for online delivery to generate COVID-19 vaccine promotion materials [36]. The BCT process begins with an educational presentation about a health topic to teams of community members, who then develop health promotion messages and dissemination plans [28]. Immunity Community trained parents to advocate for childhood vaccination and developed social marketing materials that parent advocates could distribute in their communities and share on social media [32].

Figure 1.

Ten-week codesign process and roles.

Codesign adaptations

We made two adaptations to the BCT approach. First, our curriculum included interactive activities throughout the process instead of beginning with expert presentations (Supplementary Appendix I). This built camaraderie and positioned participants as experts. Second, codesign teams were organized by self-identified race instead of being culturally diverse, and they were asked to develop content that they believed would motivate colleagues who shared the same racial background by drawing on their lived experience and knowledge about their communities. In alignment with BCT, we trained codesign cofacilitators to promote group brainstorming and discussions, ensuring content decisions were made by codesigners. Cofacilitators—who were racially concordant with codesigners—worked in teams of one researcher and one or two community cofacilitators with professional backgrounds in LTCCs or public health.

Codesign process

Ten weekly 2-h sessions were held on the Zoom videoconferencing platform between October and December 2022. Codesigners received a loaner iPad if needed to attend. Each team codesigned two products that were disseminated at LTCCs in the codesign arm from January to March 2023. Codesigners received $75 per session and an additional $150 for completion ($900 for full attendance). They also received one $25 grocery gift card per session to provide a meal. For more information about our approach, see https://kpwashingtonresearch.org/index.php/enspire-playbook.

Process evaluation design and conceptual framework

Following Holtrop et al., we used the RE-AIM framework to design an explanatory sequential mixed-methods process evaluation examining research questions relevant to our study objectives [37] (Table 1). We did not evaluate the maintenance component of the framework because codesign was not intended to be sustained.

Table 1.

RE-AIM research questions.

RE-AIM component	Research questions
Adoption by LTCCs (site level)	What proportion of the LTCCs randomized to codesign had staff participating in codesign teams?
Reach to intended participants (codesigners)	Which racial and ethnic populations were intended to participate in codesign and which populations were actually represented? How much representation of non-leadership staff—including direct care staff—was attained? How interested in codesign were non-participating staff? What could have been done to increase enrollment?
Implementation of codesign process	How consistently were the curriculum activities delivered? (Fidelity) What was the mean attendance? What were the barriers and facilitators to attendance? What types of messages did the codesigners choose to include in the materials?
Effectiveness of codesign process	To what extent did cofacilitators support active involvement in codesign? How well did codesigners work as a team to accomplish the goals of codesign? To what extent did each codesigner perceive that they contributed to the design process and products? How did participation in codesign influence codesigners’ views of the booster and their willingness to promote it?

Data collection

Data sources are described in Table 2. We collected data from codesigners and focus groups of staff who did not participate in codesign.

Table 2.

Data sources.

Data source	Description	RE-AIM component	Incentive
Baseline LTCC staff survey	Survey fielded online at all participating LTCCs to gather staff demographic data and assess attitudes about the COVID-19 vaccine	Adoption and reach	$10
Fidelity checklist	Checklist in study database listing every activity in each codesign session and measures of completion	Implementation	Not applicable
Attendance records	Checklist in study database to record attendance of all participants on each codesign team	Implementation	Not applicable
Materials produced by codesigners	Promotional posters, giveaway items, videos, and brochure created during codesign	Implementation	Not applicable
Codesigner engagement survey	Survey offered to codesigners after the codesign sessions in weeks 5, 7, and 10 to rate their experiences	Effectiveness	Not applicable
Codesigner post-codesign interviews	Interviews conducted by videoconferencing 5 to 6 months after the codesigned materials were disseminated to explore codesigner experiences during the workshop and barriers and facilitators to attendance	Implementation and effectiveness	$75
Focus groups of staff who did not enroll in codesign	Focus groups conducted on site at LTCCs 7 to 8 months after the codesigned materials were disseminated to explore staff interest in participating in codesign	Reach	$100

Baseline long-term care center staff survey

We extracted codesigner demographics from the baseline survey of staff at all participating LTCCs (Supplementary Appendix II).

Fidelity

A researcher who was not a cofacilitator assessed fidelity by viewing a random sample of 10 video-recorded codesign sessions (20% of all sessions) and noting whether each of 52 planned activities was performed. Sampling 10 unique sessions ensured representation of each codesign team twice and each session (weeks 1 to 10) once.

Codesigner engagement surveys

The survey administered after sessions 5, 7, and 10 evaluated codesign process effectiveness by asking about codesigners’ experience during the preceding sessions (e.g. I felt like my ideas were respected by my group; see Supplementary Appendix III) using a five-point Likert scale (strongly disagree, slightly disagree, I have no opinion, slightly agree, strongly agree).

Post-codesign interviews

We conducted, recorded, and transcribed 1-h interviews on Zoom 5 to 6 months after codesign to avoid any impact on codesigner participation in disseminating materials for the trial. We explored implementation and effectiveness, capturing codesigners’ perceptions of (1) barriers and facilitators to codesign attendance and (2) their experience. Interviews were conducted by racially concordant researchers.

Focus groups

Seven to eight months after materials were disseminated, staff who did not participate in codesign were invited to join 2-h, in-person focus groups through posters and emails or texts from administrators to all currently employed staff with invitations to complete a screening survey to determine eligibility. Administrators were excluded. Staff were excluded if they were not comfortable speaking English in a group. Focus groups had 5 to 12 participants (mean = 8.7) with 86 participants from Georgia and 71 from Washington. Focus groups were video- and audio-recorded at 18 LTCCs enrolled in the trial. Recordings were transcribed verbatim for analysis. Transcripts provided data about staff interest in participating in codesign.

Data analysis

Quantitative data analysis

We report summary statistics for demographic data from the staff survey, fidelity checklist, attendance records, and codesigner engagement survey.

Qualitative data analysis: interviews and focus groups

We conducted thematic analyses of interview and focus group data following the Standards for Reporting Qualitative Research [38] and processes recommended by Crabtree et al. and Creswell [39, 40]. We generated a priori codes based on the interview guide and emergent codes while reading transcripts. The full analytic team revised the interview and focus group codebooks until all agreed on definitions and applications to the text. Two coders independently coded each transcript and resolved differences by discussion to consensus. Once coding was complete, designated study team members reviewed the data and generated analytic coding memos that summarized and further developed key themes and subthemes with substantiating data [41]. Memos were discussed by the team for further analysis. We managed coded data with ATLAS.ti version 23. For illustrative quotes, “INT” indicates interview with participant number, and “FG” indicates focus group with site number.

Qualitative content analysis of codesigned materials

Following Schreier’s approach [42], we created a deductive coding frame based on the earlier 5Cs framework [43] and the more recent 7Cs framework [15–17] that describe the psychological antecedents of vaccination (e.g. ‘Complacency’, ‘Collective Responsibility’). Given our objective of describing the types of messages present in the materials, we adapted Schreier’s approach to apply binary codes to indicate (i) presence or absence of each antecedent in the text and (ii) presence or absence of two types of images: individual or collective. Using a scoring spreadsheet (Supplementary Appendix IV), two researchers independently scored the materials, then discussed the coding to reach 100% agreement.

Research team and advisory committee

Disciplines of the 16 quantitative and qualitative analytic team members include anthropology, sociology, global health, public health, health services, medicine, and biostatistics. Two community co-investigators (one administrator and one certified nursing assistant) provided LTCC employee perspectives. Four community cofacilitators and a National Advisory Committee including LTCC industry leaders, researchers, codesigners, and community co-investigators provided input throughout implementation and interpretation of findings. Many of the research team members and National Advisory Committee members have racial and ethnic backgrounds that are reflective of the codesigners’ backgrounds.

Results

Results are organized by RE-AIM topic with data sources as follows: Adoption of codesign (codesigner demographic survey data); Reach (codesigner demographic survey data, focus groups); Implementation (fidelity checklist, attendance records, codesigner interviews, content analysis of codesigned materials); and Effectiveness (engagement survey, codesigner interviews).

Long-term care center adoption

The intent was for every LTCC in the codesign arm of the trial to send one employee to each codesign team in the corresponding state. Of 20 LTCCs randomized to codesign, eight had zero codesigners. Eight LTCCs had a codesigner on only one codesign team, three LTCCs had codesigners on two teams, and one LTCC had one codesigner on all three teams in the state. Supplementary Appendix V shows the number of LTCCs represented on each team.

Reach to intended participants (codesigners)

Of 27 staff who consented, 24 participated fully and 3 withdrew without completing codesign (Fig. 2).

Figure 2.

Study flow diagram: long-term care staff screened and enrolled in codesign; the total number of staff who saw recruitment materials at the 20 participating long-term care communities was not available.

Reasons for exclusion of potential participants

Of 83 screened, 33 refused passively and 15 were excluded. Reasons for exclusion included no longer being employed at the LTCC (n = 4), being an administrator (n = 1), expressing anti-vaccine sentiments (n = 1), unwillingness to share booster promotion materials at work (n = 1), concern about the time commitment (n = 3), and passive refusal (n = 4). One person met criteria but was excluded because of insufficient volunteers for a team of their self-identified race. Of the remaining 35 who were invited to participate, three declined and five refused passively.

Codesigner characteristics

We had enough volunteers for five of the intended eight teams: three in Georgia and two in Washington, each with ≥4 codesigners. We divided Black codesigners in Georgia into two teams based on age (≤45 and >45 years) because our study team felt that age might influence communication channels used by recipients of codesign materials and the messages they would find convincing. Washington had a single Black, African American, and African immigrant team. (Hereafter we refer to teams as Black based on how teams referred to their communities in codesigned materials.) We lacked sufficient volunteers for three planned teams. No Afro-Caribbean staff completed a codesign interest survey. Only one staff member identifying as Asian or Pacific Islander consented. Of eight Latinx staff who completed a codesign interest survey, three refused passively, three were excluded due to ineligibility (one administrator and two staff who were leaving the organization), one declined for unknown reasons, and one identifying as Latinx and White chose to join a White team. Half of the codesigners held leadership roles (e.g. human resources director or infection control nurse) and half were in non-leadership roles, including five certified nursing assistants (Table 3).

Table 3.

Codesigner demographic characteristics.

	(N = 26)a
Characteristic	n	(%)b
Role  Leadershipc  Certified nursing assistant  Other non-leadership staffd	13 5 8	(50%) (19%) (31%)
Duration of employment at current employer   <1 year  1–3 years  4–10 years   >10 years  Missing	8 7 4 5 2	(31%) (27%) (15%) (19%) (8%)
Geographic location of employere  Rural  Urban/suburban	9 17	(35%) (65%)
Codesign team  Georgia Black ≤45  Georgia Black >45  Washington Black  Georgia White  Washington White	6 7 4 4 5	(23%) (27%) (15%) (15%) (19%)
Age  18–34 years  35–44 years  45–54   ≥55 years  Missing	6 4 5 8 3	(23%) (15%) (19%) (31%) (12%)
Gender identity  Woman  Man  Missing	19 4 3	(73%) (15%) (12%)
Highest level of education  Technical or vocational school  Some college, university, or community college  2-year associate degree  4-year bachelor’s degree  Post-graduate or professional degree  Missing	2 8 5 5 3 3	(8%) (31%) (19%) (19%) (12%) (12%)

^aThis excludes one person who consented but withdrew before codesign began.

^b Percentages may not add up to 100 due to rounding.

^cLeadership refers to staff in management roles, such as directors and supervisors.

^d Other non-leadership staff refers to staff such as receptionists, executive assistants, and housekeepers.

^eUrban/suburban vs. rural was defined as being inside vs. outside the largest metropolitan statistical area designations—Atlanta and Seattle.

Barriers and facilitators to enrollment perceived by staff who did not participate in codesign

Focus group participant demographic characteristics are presented in Supplementary Appendix VI. The most frequently mentioned barriers to enrollment were a lack of time or interest (Table 4, Quote 1a.1). Only a few directly stated lack of interest because of opposition to the vaccine (Quote 1b.1). However, many expressed hesitancy or anti-vaccine sentiments, which could influence their interest. A few who were vaccine-hesitant said they would still be willing to enroll (Quote 1b.2.). Suggestions to increase enrollment included scheduling codesign during the workday, holding shorter or fewer sessions, and asking codesigners to complete tasks as homework instead of meeting. Facilitators to enrollment included interest in the codesign concept and the incentive (Quotes 2a.1 and 2b.1).

Table 4.

Barriers and facilitators to enrollment perceived by staff who did not participate in codesign.

Domain	Theme	Illustrative quote
Barriers to enrolling in codesign	a. Lack of time	1a.1 It’s not a bad idea, but finding the time to do that. The time would be the hardest. Even if it’s virtual, between work and kids. (FG-2)
	b. Opposition to the vaccine	1b.1 But I would be participating in something I don’t believe in so I’m not going to advertise or help advertise something — $900 is nice though. So the people who do want to participate, I think that’s great. (FG-4)
		1b.2 I would possibly take part in a group like that just so that people who need the appropriate information and where to get [the booster], I’m still in support of that. Just because I didn’t feel like I would want to get it again doesn’t mean I don’t support anybody else who feels like they want or need it. (FG-2)
Facilitators to enrolling in codesign	a. Approach of engaging coworkers is appealing	2a.1 I do think it’s a good idea, again, because you’re utilizing people that the facility will know and then distributing that information or using their faces — hey, you should get your vaccines, making it personal, letting people know that your coworker got it. (FG-3)
	b. Incentive	2b.1 Honestly, I feel like that is a lot of money. (FG-1)

Implementation of codesign process

Fidelity

During the 10 observed sessions, 51 of 52 activities (98%) were completed; one icebreaker was omitted.

Attendance and perceptions of barriers and facilitators toattendance

Mean session attendance per codesigner was 9 of 10 (SD = 1.5, range 4–10), including the two who withdrew after five sessions. Despite high attendance, some codesigner interviewees reported barriers related to competing priorities or the evening meeting time (Supplementary Appendix VII). However, others considered the evening time a facilitator. A few codesigners in non-leadership roles that might not regularly involve online meetings mentioned technicalxs difficulties. Supplementary Appendix VIII provides interviewee demographic characteristics.

Content featured in codesigned materials

Codesigners chose many messages that aligned with the psychological antecedents of vaccination (7Cs) in the 10 codesigned materials (Fig. 3). The most commonly addressed antecedent was ‘Convenience’ (n = 9)—most often by providing a government website about accessing the boosters. (Table 5). ‘Collective Responsibility’ was addressed in all materials except giveaway items. All materials except giveaway items featured ≥1 collective image (e.g. families, staff, residents). Four featured ≥1 image of one person (Supplementary Appendix IX).

Figure 3.

Three examples of codesigned posters are provided; other codesigned products included videos, giveaway items with a booster promotion message, and a brochure with codesigners’ personal testimonials about getting the booster vaccination.

Table 5.

Presence of 7Cs psychological antecedents of vaccination in codesigned materials.

7Cs construct	Definition of code for content analysis	Illustrative quote from codesigned materials	Number of codesigned materials in which antecedent is present
Confidence in vaccine (1)	Messages promoting trust in the safety and effectiveness of the vaccine	I contracted COVID and I know, had I not been vaccinated, it could have been a totally different outcome. (Georgia >45 Black Team Brochure)	7
Confidence in vaccine (2)	Messages promoting trust in research and vaccine development	N/A	0
Confidence in systems (1)	Messages promoting trust in government and policymakers regarding vaccine policy and safety monitoring	N/A	0
Confidence in systems (2)	Messages promoting trust in health care system and health care professionals	I am thankful to work for a community that made sure that we were among some of the first health care workers to be vaccinated. (Georgia >45 Black Team Brochure)	2
Complacency (1)	Messages about how the vaccine is necessary to keep you healthy	So why should you get boosted? Family and togetherness. Health and protection against the virus. (Georgia >45 Black Team Video)	7
Complacency (2)	Messages about the importance of vaccines due to the seriousness of the pandemic	There are so many other reasons why Black people don’t exist. COVID does not need to be one. Let’s get back to the future. To find a COVID vaccine near you, go to www.vaccines.gov. (Georgia >45 Black Team Video)	3
Convenience	Messages about where or how to get the vaccine	To find a COVID-19 vaccine or booster, visit vaccines.gov or ask your supervisor. (Washington White Team Video)	9
Calculation (1)	Messages about the benefits for the individual of getting boosted	Getting boosted is an act of love. Protect yourself and those you love. (Georgia White Team Poster)	6
Calculation (2)	Messages about the risks for the individual of not getting boosted	I have a preexisting condition, lupus. So putting everything in perspective, along with the pressures of the job, I did what I thought would save my life if I were to come down with COVID: I made the decision to get vaccinated. Approximately 4 months later, I contracted COVID and I know, had I not been vaccinated, it could have been a totally different outcome. (Georgia >45 Black Team Brochure)	2
Collective responsibility (1)	Messages about the booster as a collective action to stop the pandemic	For us to help others in the movement of COVID-19 as black people and for our community: Help me to help you spread the word about boosters. (Washington Black Team Poster)	3
Collective responsibility (2)	Messages about the booster as a way to keep others from getting infected	Get knowledgeable. We are in this together. COVID does not discriminate against age, gender, mental health, physical health, nationality, social status. Picture your future—get boosted! (Georgia ≤45 Black Team Poster)	8
Social conformism (1)	Messages about how family, friends, coworkers, codesigners, or community at large support the vaccine or are getting vaccinated	I am knowledgeable for those I take care of. Get knowledgeable. I am knowledgeable for my family. Get knowledgeable. I am knowledgeable for myself. Get knowledgeable. They weren’t knowledgeable until they understood more information about the booster. Get knowledgeable. We’re in this together. Get knowledgeable. (Georgia ≤45 Black Team Video)	3
Social conformism (2)	Messages about how healthcare workers support the vaccine or are getting vaccinated	A Nurse’s Story… ‘Each booster has given me stronger mental and physical armor, more strength.’ (Georgia >45 Black Team Brochure)	3

Effectiveness of codesign process

Of 26 codesigners, 23 completed at least one of the engagement surveys at weeks 5, 7, and 10, resulting in a total of 54 completed responses (week 5: n = 22, week 7: n = 18, week 10: n = 14). Across all engagement surveys, ≥87% of respondents reported “strongly agree” on five items about engagement. At week 10, 93% reported “strongly agree” on five additional items, including: (a) products captured team’s vision, (b) happy with each product, (c) excited to share products, (d) learned a lot from participating, and (e) worth their time.

Using interview data, we explored themes in four domains: Domain 1) Facilitation, Domain 2) Teamwork, Domain 3) Codesigner perceptions of their individual contributions, and Domain 4) Personal impact related to vaccine promotion and vaccine hesitancy (Table 6). Here we report findings in these four domains related to three aspects of this study’s specific context: (1) the organization of codesign teams by self-identified race, (2) the setting of long-term care, and (3) the topic of promoting the COVID-19 vaccine. For findings related to codesign more broadly, see Supplementary Appendix X.

Table 6.

Codesigner perceptions of the effectiveness of the codesign process.

Domain	Theme	Illustrative quote
Facilitation	a. Ensuring balanced participation and consideration of different opinions	Quote 3a.1 The way that they handled this one made it so somebody like me wanted to continue to come and socialize and express their feelings toward things or express their ideas. . . . They should keep doing that. Making it a safe place for people to go to bounce ideas off of. Because for something as controversial as the COVID vaccine, it can get heated and I think they did a really good job making sure that everybody respected each other. (INT-11)
Teamwork	a. Feeling comfort and connection due to shared racial background	Quote 4a.1 We were all comfortable with each other. I guess nothing was off the table. We could all come together and talk about this issue, and us being our race, how it fitted into it. (INT-12)
		Quote 4a.2 It did play a part that all of us were African American and because we know how COVID exists for us as African Americans. We were the ones mostly dying from it. (INT-7)
		Quote 4a.3 If we would have had a mixed codesign, especially with the younger generation and the older generation, I think it would have limited some of the ideas that came out of the codesign . . . someone may have said, ‘Oh, I don’t want to say this because I don’t want them to think I’m ignorant.’ . . . So I think creating a group with age and ethnicity and everything else, it created more of a comfort zone where you would feel like you were with your peers. (INT-5)
		Quote 4a.4 I don’t always feel super comfortable talking to a wide demographic of people. Just by being an all-White group, it felt more exciting than concerning. (INT-9)
		Quote 4a.5 I was really happy with what we produced. We were Caucasian, we were mostly administrators, and this was ours. (INT-15)
Individual contributions to codesign	a. Contributing based on lived experience	Quote 5a.1 I got COVID, and it’s okay because the vaccine did its job. I wasn’t as sick as some of the people that I have heard didn’t have the vaccine. . . . I feel like I was able to speak from my personal experience of what COVID did for me. (INT-3)
	b. Feeling uncertain about contributions	Quote 5b.1 You know in every group you’ve got people who are more talkative than you and I didn’t really get a chance to really say it. So I’m not sure. I think [my contribution was] just being there as what opinion I could add. (INT-2)
		Quote 5b.2 I just said something off the top of my head that kind of helped it flow better. One of the facilitators said, ‘That’s a good saying,’ and that made me feel like I really contributed. (INT-12)
		Quote 5b.3 It took me outside my comfort zone and that made me feel good, that I could participate in something like this. (INT-12)
Personal impact related to vaccine promotion and vaccine hesitancy	a. Engaging in vaccine promotion	Quote 6a.1 The fact that all of us got to tell a little piece of our story, what we went through during COVID, whether it was that person got COVID or someone they loved got COVID. We got to tell those stories and then go back to our communities and give those pamphlets out. So again, the coworkers could see what we went through and why we got vaccinated and got boosted. If it just encouraged one or two people, that’s a lot to me, because that could save somebody’s life. (INT-8)
	b. Feeling less hesitant about the booster	Quote 6b.1 When we were all talking and putting everything together, it sold me on getting the booster shot. I didn’t just learn for myself, I learned for my family and was able to educate them as well, and I ended up getting the other two boosters. (INT-7)
		Quote 6b.2 It gave you a chance to think about COVID in a more relaxed setting . . . Because we got off work, no one was demanding or beating us with a hammer. It was just very laid back, you were able to just talk about your questions or concerns outside of the workplace. (INT-5)

Domain 1: Facilitation

While participants shared many positive impressions of facilitation, the most noteworthy theme in the context of widespread vaccine hesitancy was the effort to ensure balanced participation and consideration of different opinions. One participant described codesign as a safe place to share ideas (Quote 3a.1).

Domain 2: Teamwork

Codesigners shared appreciation of their teams and the group process. The sense of comfort and connection that codesigners attributed to having shared racial background with their teammates was a notable theme given the study design of organizing teams by self-identified race.

Several Black codesigners highlighted feeling comfortable due to their shared racial background. They described relating to each other (Quote 4a.1) and reaching consensus easily. Black codesigners also described having common ground due to the disproportionate trauma and loss experienced by Black families and communities during the pandemic (Quote 4a.2). A few linked their comfort to shared gender identity in an all-woman team and being with participants of a similar age (Quote 4a.3).

Although fewer White codesigners directly linked their comfort to shared racial background, there was evidence of greater comfort and a sense of shared identity (Quote 4a.4). One participant highlighted a sense of ownership from designing materials with codesigners who shared both racial and professional background (Quote 4a.5).

Domain 3: Individual contributions to codesign

Nearly all codesigners readily highlighted skills, experiences, or ideas they contributed to codesign. A few described providing the perspective of having COVID-19 or knowing someone who had (Quote 5a.1).

However, a few described speaking less frequently or expressed uncertainty about their contributions (Quote 5b.1). Another codesigner who mentioned being “quiet” initially described the impact of a facilitator validating their idea (Quote 5b.2) and ultimately described the codesign experience as positive (Quote 5b.3). Most codesigners who described hesitation to speak had professional roles with lower wages and less power in the LTCC organizational structure.

Domain 4: Personal impact related to vaccine promotion and vaccine hesitancy

Some codesigners described engaging in vaccine promotion (Quote 6a.1) and getting the booster themselves as a result of participating (Quote 6b.1). Although the intent of the study was to promote the booster to their colleagues, codesigners also reported persuading LTCC residents, family, and friends (Quote 6b.1). Codesign was not intended to influence personal vaccine hesitancy, but the discussions may have helped address concerns (Quotes 6b.2). Codesigners who were already convinced of the vaccine’s importance reported no change in their opinions.

Discussion

We evaluated a codesign process with LTCC staff that generated COVID-19 booster promotion materials for a randomized trial comparing them to generic materials. We identified evidence of challenges in the codesign process with adoption and reach, and successes with implementation and effectiveness. The primary evidence of challenges with codesign adoption by LTCCs was that we could not enroll codesigners from all LTCCs randomized to codesign in the trial. Challenges with reach included the disproportionately low representation of direct care and other non-leadership staff in codesign teams and the lack of representation of all intended racial and ethnic groups. The main barriers to enrollment were limited time or interest.

Evidence of successes related to implementation included high attendance, high fidelity to planned session activities, and creation of materials that reflected known psychological antecedents of vaccination. The effectiveness of the codesign process was evident in codesigners’ perceptions that they contributed individually to the materials, which is an important measure of success given the goal of producing participant-designed materials. A few codesigners reported promoting the booster in the workplace and beyond. Although codesign was not designed to encourage codesigners to get vaccinated, a few got the booster as a result of participating. The effectiveness of the process was also evident in codesigners’ positive impressions of facilitation and their sense of connection with their teams.

Our approach of organizing teams by self-identified race to tailor materials—which differs from other codesign efforts to support COVID-19 vaccination [36, 44–47]—led to insights about how shared racial background led to a sense of common ground. In addition to race, codesigners on Black teams linked shared age to greater comfort participating. Shared gender identity was also described as supporting comfort on the ≤45 Black team that unexpectedly included only women. On White teams, shared race and professional background were linked to shared identity and a sense of ownership of the codesigned materials.

Reaching a broad pool of codesigners with varied perspectives on vaccines is critical considering that the development of an effective codesigned intervention requires engaging participants who are representative of the intended population [48]. Our findings underscore the importance of engaging vaccine-hesitant staff and addressing structural barriers (e.g. time commitment) that disproportionately affect direct care staff. We found that a few staff would have been willing to participate despite vaccine hesitancy. To include a broader range of perspectives, in-person outreach could encourage staff with doubts about the vaccine to participate. Considering that vaccine-hesitant participants have dominated discussions in other COVID-19 vaccine codesign efforts [44], screening is still needed to ensure codesigners are willing to support codesign goals. Our findings also suggest that codesign with LTCCs should consider condensed processes (e.g. fewer meetings, a 1-day workshop, a few half-day workshops) to be accessible for more staff. Adding focus groups to review draft materials during codesign could provide additional input from staff who cannot commit to codesign.

Regarding the effectiveness of the codesign process, our findings were consistent with other codesign studies [48–51] that found that codesigners valued cofacilitators’ efforts to share power, build trust, and create a comfortable environment. Despite the overall positive findings, a few codesigners in roles with lower wages and less power in long-term care described participating less frequently or expressed self-doubt about their contributions. The codesign literature describes power-sharing strategies for teams with service users (e.g. patients) and service providers (e.g. clinicians) [51–53], but the potential impact of power differentials in professional hierarchies must also be considered. Although online collaboration tools (e.g. for anonymous voting) are recommended for supporting participation [49], our findings suggest that technical support is needed to ensure equitable access, which is consistent with other studies [45]. Forming codesign teams or organizing breakout discussions among codesigners with similar professional roles could also increase comfort and participation.

Considering the role that shared characteristics such as age, gender, race, and profession play in developing cohesion and comfort on codesign teams, it is also important to consider the range of life experiences and power differentials within communities when determining team composition. Input from community partners should guide decisions about team composition to achieve participatory processes that produce high engagement.

Our process evaluation also offers lessons related to codesigning materials to promote the COVID-19 booster. In our codesign process, codesigners provided messages and images to communications professionals who then drafted the materials, which is similar to other codesign efforts promoting uptake of the COVID-19 vaccine [36, 45, 46] or developing online training [44] or scripts for virtual simulation games [47] to train health care professionals on vaccine promotion. However, the role of communications professionals in our codesign was limited to providing the technical assistance needed to draft the materials based on specifications provided by the codesigners. We chose not to provide codesigners with any orientation to vaccine hesitancy frameworks or health promotion theory.

Because codesigners had full autonomy to make all choices about content, our findings provide unique insights about the messages that LTCC staff believed would be effective in their workplace. Through retrospective analysis, we found they had developed messages that aligned with the 7Cs antecedents of vaccination, focusing mainly on ‘Convenience’, ‘Collective Responsibility’, ‘Confidence in the vaccine’, and ‘Complacency’. Conversely, they did not choose to promote ‘Confidence’ in research, vaccine development, or the government. Future research could determine which of the 7Cs might be most important to target in vaccine promotion materials for this population. Future codesign efforts could explore providing an orientation to the 7Cs or other vaccine hesitancy frameworks and the opportunity to consider selecting specific constructs to address. Such an approach could sharpen the focus of the materials and potentially enhance their effectiveness while preserving the codesigners’ autonomy when developing the content.

Limitations

Our evaluation had a few limitations. We could not determine whether codesigners’ racial and ethnic backgrounds were representative of staff at LTCCs randomized to codesign because LTCCs did not routinely collect these data. The reach evaluation could not calculate the proportion who participated in codesign relative to the number invited because recruitment included flyers in public spaces in the LTCCs. Finally, we did not interview two people who withdrew from codesign after five sessions and four codesigners who could not be reached.

Conclusion

Our process evaluation resulted in lessons for reducing barriers to joining codesign processes and supporting equitable participation. A longer duration of codesign provides more opportunity for collaboration, but a smaller time commitment or condensed format might make codesign feasible for more people and increase representativeness. To expand the pool of potential codesigners and the diversity of opinions, inviting codesigners with divergent views who are willing to contribute to the codesign goal could be useful. Strategies to support equitable participation include designing activities that support the participation of less vocal codesigners and forming teams based on shared characteristics (e.g. race, age, gender identity) to increase comfort. Decisions about team structure should be guided by community input and informed by the goal of the codesign process, the intended audience for materials or programs, and power differentials that influence codesigner comfort. Overall, we found that codesigners were highly engaged and developed materials addressing known antecedents of vaccination. Future codesign efforts could explore inviting codesigners to consider which antecedents of vaccination seem most relevant for their colleagues to determine the focus of their messages. Providing that barriers to enrollment are addressed, codesign holds promise to design booster promotion materials or other educational initiatives, such as online training or virtual simulation games, for LTCC settings.

Author contributions

Kimberly C. Arthur (Conceptualization, Formal analysis, Investigation, Methodology, Visualization, Writing—original draft, Writing—review & editing), Kanetha B. Wilson (Conceptualization, Formal analysis, Investigation, Methodology, Writing—review & editing), Breana Berry (Conceptualization, Formal analysis, Investigation, Writing—review & editing), Brianna Binion (Visualization, Writing—review & editing), Marla Becker (Conceptualization, Formal analysis, Writing—review & editing), Alphonse Derus (Conceptualization, Data curation, Formal analysis, Software, Visualization, Writing—review & editing), Sanaa L. Diop (Conceptualization, Investigation, Writing—review & editing), Margaret Gachuiri (Conceptualization, Formal analysis, Writing—review & editing), Beverly B. Green (Conceptualization, Formal analysis, Methodology, Writing—review & editing), Ahoua Koné (Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Writing—review & editing), James Licitra (Conceptualization, Writing—review & editing), Caroline Liou (Conceptualization, Writing—review & editing), Courtney E. McCracken (Conceptualization, Investigation, Methodology, Writing—review & editing), Lauren E. Nisotel (Conceptualization, Formal analysis, Investigation, Writing—review & editing), Shanise E. Owens (Conceptualization, Investigation, Writing—review & editing), Annalisa V. Piccorelli (Conceptualization, Writing—review & editing), Sophie Ramsey (Conceptualization, Writing—review & editing), Lindsay B. Schwartz (Conceptualization, Writing—review & editing), Kirsten Senturia (Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Writing—review & editing), Jennifer Svoboda (Conceptualization, Writing—review & editing), Jaclyn Volney (Conceptualization, Formal analysis, Investigation, Methodology, Writing—review & editing), Brian D. Williamson (Conceptualization, Formal analysis, Investigation, Writing—review & editing), and Clarissa Hsu (Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Supervision, Writing—review & editing)

Conflict of interest: None declared.

Funding

This work was supported by a Patient-Centered Outcomes Research Institute (PCORI) Award [COVID-2021C2-13168].

The statements in this article are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or Methodology Committee.