Scale, Scope, Speed: Reflections on a Multi-site Covid-19 Study

Abstract

Designers have a unique role to play in public health, but their involvement requires an examination their practices and methods for their fit with this new context. This article reflects on the experiences of a multi-site design team collaborating across the US and Canada to explore early-stage Covid-19 patient recovery experiences. A unique feature of this project is that it was conceived of, led by, and executed by designers situated in health systems and health research units working in diverse geographies to jointly investigate a public health phenomenon at a broad scale. We discuss three challenges to design practice encountered in this context—scale, scope, and speed. Lastly, we draw from the design teams’ cross-sector expertise to pose key questions for design as it migrates to the public health sector.

Introduction

Public health has as its goal to “promote greater health and well-being in a sustainable way, while strengthening integrated public health services and reducing inequalities.”1 The Covid-19 pandemic demonstrates that public health challenges may be “complex, dynamic, and context-specific, and can at times arise quickly and unpredictably, raising the urgency for rapid and efficient responses.”2 The field of design is characterized by a history of knowledge related to people-centered approaches to system methods for change. These may contribute to the mission of public health and potentially accelerate public health responses. This potential is increasingly recognized and even promoted by prominent public health funders, researchers, medical educators, and healthcare institutions.3 As a result, a small but growing subset of designers has gravitated to public health and healthcare, finding purpose and new opportunities for application and impact.

In May of 2020, nine such designers and individuals with public health experience convened to explore the recovery experience of patients who tested positive to Covid-19. The pandemic at that time was in its earliest phase, and little was known about the trajectory and treatment of infected individuals. While health systems were concentrating on the most acute cases (those requiring hospitalization or ICU-level care) individuals who did not meet admission standards were sent home to recover in the community. The team, distributed across five hospitals and health systems across North America, noted growing reports of community-recovering patients struggling with scant public health guidance for self-care or for coherent strategies to avoid infecting their loved ones. As a result, these individuals were assembling their resources, modifying their own behaviors, and engaging their own strategies for their recovery. Public health institutions might learn from such inventive efforts to inform future infrastructure and public health investments. To examine such community-level experiences, our team conceived a multi-site study to explore the Covid-19 recovery experience after patients had been tested and diagnosed as positive. We documented patients’ strategies; study methods and results were published in June 2022.4

This article is not a case study, but a reflection on the study team’s experience of applying a design-led approach to a rapidly unfolding public health crisis, while working cross-site and within a traditional health research paradigm. We discuss three challenging dimensions of public health research encountered in this effort—scale, scope, and speed—and the effects on the study team’s process. Lastly, we combine learnings from this project with other experiences of the study team to offer recommendations for the design field.

The Design Experience of Working on a Multi-site Public Health Study

Study team members met while working through a volunteer network of doctors and designers called the Emergency Design Collective (EDC). The Collective was launched by a design-trained trauma surgeon and a panel of design practitioners in May 2020 to generate rapid, creative responses to the growing number of pandemic challenges that healthcare and public health institutions were slow to address (for example, shortages of protective gear, drive-through testing services, drops in blood donations). The Collective created conditions where design-led teams could address diverse aspects of a public health emergency from a design perspective. Notably, the designers could define the research problem, rather than the typical approach where academic researchers and public health leaders would take the lead.

Our study team consisted of designers and health researchers employed by healthcare organizations or health-related university research units across North America: Toronto, Michigan, Chicago, San Francisco, and Los Angeles. Expertise varied: six team members had advanced design degrees; one had a Master of Public Health. Sector expertise also varied: three team members brought multi-sector design experience from industry, healthcare, and public health; six team members had experience in two of the three sectors. In the initial meeting we noted that ad hoc discussions with healthcare colleagues identified that little formal inquiry was being conducted into what Covid-19-positive individuals and families required to manage their physical and emotional needs after receiving a positive test result. Basic self-management information, such as self-care routines, when to re-test, a recovery timeline, and how to manage extended isolation, were missing. They cited these as a source of often overwhelming frustration and anxiety for Covid-19 patients. The team concluded that this presented an opportunity for a design-led exploration.

Over the course of 15 months, the team met virtually to plan and conduct its five-site study. Prior to this study, no team member had participated in a multi-site study composed solely of design-oriented team members using design processes. Multi-site studies are the norm in health services research, and several team members had experience with multi-site clinical trials and studies. The peer-reviewed literature contains no documentation of such an approach being applied by designers, suggesting this effort may be the first of its kind.

The Collective developed study protocols and interview guides collaboratively and employed data collection practices familiar to everyone in the group, such as journey-mapping. In contrast to traditional public health research, we included generative techniques such as How Might We statements to ensure translation of insights into actionable recommendations. We submitted study protocol and interview guides to be reviewed by site-level institutional research boards. We recruited two participant cohorts (community-recovered and hospitalized Covid-19 patients) from each site’s hospital service area. We used site-specific practices to identify and contact potentially eligible participants. Some site teams accessed local Covid-19 patient registries and conducted direct outreach (active recruitment); others posted flyers in clinics (passive recruitment).

Each team conducted their own interviews, managed data collection, and performed analyses. This protected participant confidentiality as well as ensuring the findings of each site were not influenced by other site data. Once all sites had concluded their work cross-site data analysis commenced. The entire team discussed the de-identified site data, using remote working tools such as Google sheets and Mural to compare such data. The entire team reviewed and clustered thematic codes over a number of sessions. Working in cross-site pairs, team members generated Insight and How Might We statements for each cluster, followed by group review and approval. A subset of team members translated data and findings into visualizations and frameworks.

At project completion, all team members convened to reflect on the strengths and weaknesses of adopting a public health research framework for design work. One team member conducted individual interviews with site members. We did this to minimize possible influence from other site members. We wanted to gain individual perspectives on three things: the overall experience of merging a multi-site research paradigm with a design-led approach; challenges to design processes posed by cross-site work and therefore institution-specific requirements; and, effects of cross-site work on data and findings.

Lessons and Challenges

The results of debrief interviews highlighted the benefits of engaging a peer network with shared values and mission, with the added benefit of different healthcare expertise, roles, and experiences. Experienced representation from user research, systems design and communication design, for example, promoted rigorous discussion and application of design methods throughout the study. Team members employed in medical research units reported that the freedom to step outside of expected qualitative practices such as grounded theory and engage in more interpretive analyses of participant behaviors, attitudes, and needs for ideation was liberating.

Across sites, team feedback pointed to three challenges for design practice in public health.

Challenge 1: How to Explore Human Experience on a Larger Scale

Public health involves identifying and addressing gaps at a population level. The ability to engage in broader sampling is critical to public health efforts, where generalizability of findings is paramount. Historically, design methods were developed to optimize solutions for a specific context and address the needs of specified users.5 Design has not been optimized to address population-level challenges requiring inputs from diverse users in diverse settings and generate findings relevant to many more individuals.

Given our population level challenges focus, the project benefitted from a multi-site structure because it enabled the study team to access geographically disparate care settings and patient recruitment infrastructure across North America. Even during Covid-19, teams housed in multiple sites created access to clinics, emergency departments, and providers that yielded a number of diverse and qualified participants who would have otherwise been difficult to recruit and verify. Single-site studies often over-index on variables that bind findings and solutions to the local context.6 Our multi-site study supported a collection and examination of local and cross-site phenomena with the larger number that public health research requires, adding credibility and confidence without site-level burden (Figure 1 ). We would argue that as designers seek to create evidence for and uptake of public health solutions, there will be a growing need to work like medical researchers and establish a peer network that is willing and trained to engage this way.

Figure 1.

Multi-site collaboration increased the scale of participant engagement to reach desirable samples of both patient cohorts, without undue site burden. © 2022 the authors.

Challenge 2: How to Manage a Larger Scope of Inquiry

If the challenge of scale is the size and reach of a given public health problem, the challenge of scope is the process and range of activities required to address the need and still produce credible output. When designing for industry and healthcare, it is standard practice to expand the margins of a given problem and incorporate a broader range of human experience and real-world behaviors for consideration.7 The advantage of this approach is that it improves the accuracy of problem-framing and challenges existing biases about the cause of, or fixes to, various problems. The disadvantage in relation to public health is that the scope of any project already assumes multiple geographies, populations, cultures, or care settings. To expand the scope of inquiry at each site further, therefore, can result in a geometric pile-up of local, contextual variables. For example, the “go wide” approach of design might explore multiple dimensions of a local population’s experience, including the patient journey through a local health system, the mental models and expectations of patients for their local health system, community conceptions of health, and workarounds or alternative strategies employed by local patients. These dimensions may collect rich, intimate insights that are productive for local problem-solving, but less relevant and overly detailed for multi-site endeavors and solutions.

To avoid geometric pile-up for both feasibility and validity, it was critical for the team to define the data and variables. Rather than inquiring about participant experiences at predefined time points in their recovery process, for example, a framework was added to collect four types of experiential data consistently at each time point (AAAA framework: Activities, Anxieties, Ambitions and Attitudes8 ). This more targeted approach improved feasibility of data collection and analysis in the time permitted. This approach also improved validity because it enabled the study team to collect highly comparable data across cohorts and locations, supporting pattern detection during analysis. While modifications of this kind were effective from a public health perspective, they lost the advantage of design’s preferences for wide-angle data collection. More consideration of this topic is warranted.

Challenge 3: How to Respond to Changes in Speed

The institutional review board requirements and review is a slow process which sat at odds with the rapid progression of the pandemic and the resulting need to engage patients living with Covid-19 at specific points in their recovery. While others have documented the shortcomings of institutional review board responses to the unique demands of pandemic-related studies,9 our study team experienced numerous barriers to conducting in-the-moment research. This included the need to interface with a number of institutional review boards, each with different local processes, which created significant variances in study implementation. Examples of local variances included how to identify patients who had Covid-19 accurately; staff participation in outreach and recruitment; and limited availability of design staff tasked with other work related to the pandemic. While the concept of the single-site, centralized institutional review board is emerging in medical research, this idea is in its early stage. The current cost, speed, and variability of single-site institutional review board processes are equally difficult to navigate for design-led projects as they are for site-specific institutional review boards in other fields. A second barrier was that site-level institutional review boards subjected the same protocol to different levels of review (Table1 ), creating significant variability and delays across sites. A third barrier was that site-level institutional review board and other institutional processes slowed the rapid experimentation of design. The study teams’ various institutional review boards, for example, required that prototypes be submitted in advance to gain permission to do the design work intended to develop them. This mismatch between design practice and research practice has been documented by others. Alessandra Bazzano and her colleagues note:

“Many of the central tenets of design thinking research, like iteration, tolerance for ambiguity, pivots, and rapid prototyping, are inherently at odds with some prevailing processes in health and biomedicine, particularly public health, where hypothesis-driven research is the norm and where the evidence base (typically the peer-reviewed literature) is used to generate concepts for study.”10

Table 1.

Differences in site institutional review board approval processes and timeframes.

Institution	Date submitted	Date approved	Days to approve	Review
UCLA	06/23/2020	09/08/2020	77	Expedited
NYGH	06/16/2020	09/03/2020	79	Full Board
UCSF	06/05/2020	08/19/2020	75	Expedited
Rush	01/29/2021	02/10/2021	10	Expedited
UMich	06/02/2020	06/04/2020	2	Exempt

In sum, while multi-site explorations are effective at addressing issues of scale, they incur costs in relation to speed. One site anticipated this and engaged its institutional review board in advance of submission to seek advice, which translated into a shortened review period. We suggest that design-led processes need to anticipate these complexities and develop timelines and project management approaches that can absorb them. In the project debrief, the study team discussed a more systematic solution: How might design evolve effective study protocols approved by an institutional review board that incorporate rapid experimentation? Members agreed this would be a critical enabler that could be developed via a cross-site consortium of designers collectively prototyping, testing and then disseminating strategies approved by institutional review boards.

Implications for Design Practice

The study team brought together design expertise from industry, academia, healthcare delivery organizations and community health. Together we applied design methods to one of the largest public health challenges in the last century. We experienced three challenges in this public health research effort—scale, scope, and speed—that differed significantly from the team’s collective experiences in the private and healthcare sectors. In response, we adapted and negotiated our design approach to function within a public health research framework and meet its expectations for scientific rigor and evidence. Specific adaptations included the need and means for broader sampling of human experience to respond to the wide scale of the Covid-19 experience; a development strategy to calibrate the scope of design inquiry to bring feasibility, validity, and productivity to the work; and project management to help align the speeds of design, institutional oversight, and the public health problem. We were not uniformly successful in these adaptations, and a number of study team members reported dissatisfaction with the changes in design practice. For example, scale and scope, coupled with the need to produce credible results, focused the team on establishing process guardrails and process monitoring (keeping sites aligned), and reduced time spent exploring and defining a design perspective (specifying key questions to answer and building a case-informed analytic lens). We would balance this differently in the future. A second example is that multi-site, multi-team analysis processes required operating at a higher level of abstraction, which obscured distinctive experiences or patient stories and diminished empathy with patients as individuals. This also made it difficult to retain and incorporate emotional aspects of the patient experience into findings. Other dissatisfiers included: the need for an environment for action—we did not have health system and public health leaders with bandwidth to act on this work; project management and the time demands on the design team were greater than in single site processes; while we succeeded in collecting diverse patient perspectives, we lacked full representation of other stakeholders, notably frontline clinicians overwhelmed with patient care. This last element was not a process decision but a real-world workforce limitation of the pandemic. Interestingly, similar limitations of design practice were reported by Kara Durski and her colleagues, when applying design (specifically design thinking) to the 2014 Ebola outbreak.11

In post-study discussions, our team debated the importance of our experiences with scale, scope, and speed and probed for generalizability to design practice more broadly. We offer these discussions to the design community because design practices are receiving increased attention from healthcare and public health institutions and leaders.12 We note two forms of attention, both of which imply design will encounter new pressures as invitations to participate grow.

The first form of attention is the uptake of and experimentation with design processes among key gatekeepers of healthcare and public health: funders, researchers, and healthcare institutions. These gatekeepers have expressed cautious optimism that the people-centered paradigm of design may offer a missing and complementary perspective to inform health interventions. Federally funded research agencies in the US, such as the Agency for Healthcare Research and Quality (AHRQ) and the Patient Centered Outcomes Research Institute (PCORI), have requested proposals that incorporate design methods.13 Conservative US research institutions—gatekeepers of practices such as Harvard’s T. H. Chan School of Public Health, the University of Illinois at Chicago’s Population Health Sciences Program, and the University of Michigan’s Institute for Clinical and Health Research—have established design labs or employed designers to participate in clinical trials and broader research efforts.14 In the healthcare sectors of many nations, including the United States, the United Kingdom, Australia, and other countries, healthcare systems have established design capacities that are diversely placed and tasked.15 These systems focus on many topics, including healthcare strategy, quality improvement, service line development, and patient experience. Two of the largest philanthropic public health organizations in the world, the US-based Bill and Melinda Gates Foundation and the Robert Wood Johnson Foundation, both promote human-centered design (HCD) as a critical tool for addressing public health challenges.16 Applications of design methods by researchers and health system faculty have grown significantly in the last ten years and disseminated through peer-reviewed publications. Among PubMed-indexed medical journals, over 50% of peer-reviewed publications that incorporate design methods have been published since 2017 (Figure 2 ). In sum, three powerful gatekeepers in healthcare and public health—funders, publishers, and healthcare systems—indicate that design is increasingly being examined for its ability to improve the performance of healthcare and contribute to the health of the world’s populations.

Figure 2.

The number of peer-reviewed publications with human-centered design are increasing, with 54% published in 2017 or later. Search terms include: human-centered design, user-centered design, human-centred design, user-centred design, or design thinking. The search used the U.S. Library of Medicine’s PubMed.gov website up to January 1, 2022. © 2022 the authors.

A second form of attention targets the results of this uptake of design in public health and healthcare. Here the scientific community is asking harder questions: Where is design having the most impact? What is the evidence of design’s impact? Is that effect replicable? Systematic reviews, considered a benchmark source for establishing evidence in the medical sciences, applies specific reproducible methods to synthesize peer-reviewed publications on a given research topic systematically to find patterns in outcomes.17 Two recent systematic reviews sought to identify evidence for the impact of design methods in health. A narrative review18 and a scoping review19 both noted the difficulty of finding papers that met inclusion criteria. Both concluded that a lack of standardization in the description of design, its core concepts, and its methods, presented barriers to assessing design’s impact and therefore to formalizing knowledge about its application. There is a number of potential explanatory factors for these weaknesses. This includes the design expertise of those who participated in the work (participation of trained designers was not an inclusion criterion for either review). Nonetheless, these publications highlight potential limitations of design methods for health sectors.

A question arises in the design field about whether shifts in design practice are needed to function effectively in public health. While championing design’s role in public health, both Robert Fabricant20 and Panthea Lee21 asked whether design practices that prioritize experimentation, reinvention, end-users, and market impact are sufficient for public health without adaptation. Their proposed correctives target process modifications and shifts in design focus. For example, designers have been encouraged to: incorporate more effectively with existing public health research and development processes;22 find ways to integrate design insight into other sources of public health expertise that produce value, such as macroeconomic, behavioral economic, and political economy analysis;23 shift focus from products and projects to systems and outputs that drive policy change, and thereby expand their user focus to include public health decision-makers and development investors.24

Considering this design commentary, recent medical literature, and the collective experience of our study team in health and public, we suggest there is work to be done to integrate design more fully with the public health sector. We would like to contribute the following questions for consideration.

How might design promote standardization of the vocabulary, definitions, and practices of its methods?

As the recent systematic reviews highlight, if design is to flourish in the health and public health spaces, it must formalize its processes and vocabulary better. Codifying tools and methods to become accepted as valid in the healthcare sector is critical to enabling their meaningful use. Medicine and public health—scientific- and academically-oriented fields—have rigorous and shared terminology that is assimilated during education and reinforced in publication. As a practice-led, rather than academic-led field, design has accumulated localized practices and variations in how it defines key terms.25 Differences between a “journey map” and an “experience map,” for example, continue to be defined and depend on who’s asked.26 To participate in public health, design must strive to formalize its own practices and terminology within its own field first, so that it can orient and engage external stakeholders with confidence. Design educators and journals are essential to this effort.

How might design research evolve to fit a broader research paradigm?

Design research is largely a borrowed science, with practices deriving from systems methodology and the social sciences. While the adoption of methods from adjacent fields is productive for design, this process should be robustly documented and pursued with rigor and attention to the original methods. This will help design research meet an important criterion: that when similar methods are used in a similar context there will be a creation of insights and results that are relatively commensurate, consistent, and reliable. Such documentation and definition will further enable the large-scale collaborations and impact-oriented projects that characterize public health. To do this will also help designers more successfully describe their methods to Institutional Review Boards and in research publications.

How might design think differently (and rigorously) about the effects of scale when applying core practices?

In the introduction, we outlined the advantages and disadvantages of design research in the health care sector. In particular, we pointed out that some core design practices become more difficult at scale. How do we model a user in public health, for example, when constituents are likely not a homogenous group? 300,000 “end-users” of a commercial product, for example, might all be defined by their common use of the product. 300,000 stakeholders in a healthcare system might all defined by their use of a particular hospital. Such groups share enough similarities (for example, engaging in similar activities, or employed by the same types of organizations) so that subgroups and design requirements can be identified. A similarly sized population of 300,000 people may not share much in the way of internal characteristics. How, then, are we to model users at a population level? What are the tools that will help us see common needs and manage design requirements? Prototyping faces similar challenges at scale: how do we prototype a population-level intervention? What are we testing? How many people will need to be involved for us to know if we have learned from the prototype? These and other core practices need examination and thoughtful adaptation to serve the public health context.

How can design engage seriously with the need for evidence in healthcare and population health?

Healthcare and public health are science-based, evidence-driven fields where standards for evidence are more tightly specified. In healthcare, evidence is a multi-construct concept that includes “best research evidence, clinical expertise, patient’s morals, values, and beliefs and information from the practice context” and based on multiple sources.27 The agile and speculative design processes have a role to play but may not be able to generate the volume or quality of evidence required to justify adoption of new solutions. Design as a field of practice would do well to establish empirical evidence for its methods of problem identification, solution generation, and impact measurement, as well as for the generalized knowledge about design in healthcare. The precise nature of this evidence is unclear and an interesting point for development. However, accumulating evidence and establishing a science of design methods over time is important to driving confidence in designed artifacts and services; the practice of designers; and eventually in any policies or services that are based in design research. Building confidence in design evidence may also increase inclusion of rapid, iterative design approaches in situations—such as public health emergencies—when those methods might add unique value.

How might we encourage designers to cultivate “know about” in addition to “know how”?

It is important for designers to cultivate curiosity about the many relevant and more influential specialties that are integral to healthcare and public health efforts. Program evaluation, statistical analysis, implementation science, health economics, and epidemiology are just some of the fields with which design knowledge will need to interoperate during development and implementation. It is not critical that designers become experts in these fields. However, designers should be conversant in the terminology. The ability to speak and understand the terms will help designers to work more effectively. This will sharpen and accelerate the integration and impact of design.

Conclusion

Our study team applied design methods to the experience of patients living with and recovering from Covid-19. The team’s unique experience as designers from across North America cooperating to produce knowledge about pandemic recovery also led to the discovery of critical knowledge for designers wishing to practice or participate in public health research. Our experience suggests that designers appear well-positioned to contribute to public health efforts to address the needs of their communities, to help craft solutions to deal with those needs, and to implement and scale solutions to support population health. To accelerate the integration of design and the public health sector, we have in this article offered several areas to consider for improvement. These include data collection and research framing. These need to work across large populations and still produce the contextual information that has been core to design’s success in producing fit-for-purpose solutions. Additionally, development of a peer network of designers trained and willing to work in multi-site collaborations would enable the field to address the demands of scale more effectively in public health initiatives. Of critical importance is developing new strategies for navigating institutional practices that may slow the speed of working together but protect the ethical considerations inherent in these efforts. Designers would do well, for example, to prototype collectively and share institutional review board submission strategies that preserve the space to practice core design methods while meeting requirements for oversight. Lastly, we argue that there is a need for academic reflection on design practices to produce a workforce capable of practicing in an intensely scientific sector like public health better. Progress across all these areas will enable design to contribute to public health and create a base of theory and practice that permits designers to thrive in public health settings.

Declaration of Interests

There are no conflicts of interest involved in this article.