Skip to main content
NCBI home page
EPA Author Manuscripts logoLink to EPA Author Manuscripts
. Author manuscript; available in PMC: 2026 Mar 11.
Published in final edited form as: Community Sci. 2023 Jun;2(2):e2022CSJ000021. doi: 10.1029/2022CSJ000021

What Does Equitable Co-Production Entail? Three Perspectives

K L Akerlof 1, K M F Timm 2, A Chase 1, E T Cloyd 3, E Heath 4, B A McGhghy 5, A Bamzai-Dodson 6, G Bogard 4, S Carter 7, J Garron 2, M Gavazzi 8, N Kettle 2, M Labriole 9, J S Littell 10, M Madajewicz 11,12, J Reyes 13, L Rivers III 14, J L Sheats 15,16, C F Simpson 17, R C Toohey 10
PMCID: PMC12973253  NIHMSID: NIHMS2131326  PMID: 41816204

Abstract

Co-production practices are increasingly being adopted in research conducted for the purpose of societal impact. However, the ways in which co-production is conducted can perpetuate long-standing inequity and inequality. This study investigates which principles of co-production design are perceived to advance more equitable processes and outcomes based on the experiences of participants in three projects funded by U.S. federal programs that support decision-relevant climate science, along with others engaged in co-production efforts. We found three distinct perspectives: (a) Ways of Knowing and Power; (b) Participants and Interactions; and (c) Science as Capacity Building. Each viewpoint differentially weights the salience of statements associated with five dimensions of co-production practices: (a) outcomes; (b) power; (c) place-based, community rights and respect; (d) audiences and participation; and (e) interactions. In the final stage of the study, we hosted a workshop of participants representing various roles in co-production efforts to vet and discuss each perspective. We found that the perspectives remained distinct after each of the groups selected core statements that reflect their views. The degree of variation across the three perspectives suggests that co-production processes would benefit from an initial discussion of, and decisions about, rules of engagement to ensure that participants view the process as equitable.

1. Introduction

Unlike basic science, studies conducted for the purpose of societal decision-making require the involvement of broad publics (Gibbons, 2000). The U.S. Global Change Research Program (USGCRP), mandated by Congress in 1990 to provide decision-relevant climate science, spurred the establishment of some of the longest-running and most geographically diverse programs to conduct these types of applied research. The National Oceanic and Atmospheric Administration (NOAA) Climate Adaptation Partnerships (CAP), Department of the Interior (DOI) Climate Adaptation Science Centers (CASCs), and U.S. Department of Agriculture (USDA) Climate Hubs (Pulwarty et al., 2009; USGCRP, 2023) represent some of the most prominent examples. These federal programs may shed light on an issue that is increasingly recognized as a significant challenge to decision-relevant scientific research: the involvement of people who historically have been underserved by government programs and/or have experienced discrimination and exclusion (Early & Kotzebue, 2021). Public administration and political science scholars have long warned that conditions of social inequity and inequality may worsen if the quality of governance and public services is dependent on community participation (Einstein et al., 2019, 2022; Rosentraub & Sharp, 1981). When the voices of marginalized groups are unlikely to be heard and to influence policy outcomes, deliberative democratic ideals for participatory processes may conflict with values of diversity and equity (Fainstein, 2014). This raises the question, what does equitable co-production of climate research entail?

This question is particularly pertinent due to federal plans to direct at least 40% of benefits from energy and climate investments to underserved communities (Young et al., 2021) and the increasing expectation that researchers will conduct science that has direct societal impact. Over the last 20 years, support for basic research—”experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundations of phenomena and observable facts, without any particular application or use in view” (NCSES, 2022, p. 4)—has dropped from 58% to 25% of total federal research and development spending (National Science Foundation, 2020, 2022). By 2020, the vast majority (75%) of federal allocations were directed toward applied research and experimental development, drawing on scientific and other forms of knowledge to produce or improve products and processes. Within these forms of science lies a subcategory of research that is transdisciplinary, produced for a specific application, conducted by diverse groups of people, socially accountable, and held to quality standards defined by its users—not those of scientists alone (Gibbons et al., 1994). Sometimes referred to as “Mode 2” research, similar descriptions of this complex relationship between knowledge production and diverse publics appear within a wide array of academic literature spanning domains of policy and practice, but under different names (Fransman, 2018). Examples include participatory research (Hall, 1992; Minkler, 2010), action research (Lewin, 1946), civic science (Clark & Illman, 2001), community science (Wandersman, 2003), post-normal science (Funtowicz & Ravetz, 1993), and co-production (Jasanoff, 2004; Lemos & Morehouse, 2005).

1.1. Co-Production

Increasingly federal agencies and academics are using the term co-production to describe engagement with people and forms of knowledge outside the bounds of their research institutions (Bremer & Meisch, 2017; Lemos et al., 2018). The process is generically described as one that “brings together diverse groups to iteratively create new knowledge and practices,” whether to generate actionable knowledge or spur the redistribution of power and societal transformation (Jagannathan et al., 2020, p. 22). But co-production definitions somewhat differ across the fields of public administration, sustainability science, and science and technology studies (STS)—all of which employ the phrase (Wyborn et al., 2019). STS theorize that co-production emerges organically from the intersections of science and governance (Jasanoff, 2004). Sustainability and public administration disciplines take a normative approach, conceptualizing co-production as an intervention enacted for the purposes of improving respectively science and policy (Lemos & Morehouse, 2005), and public services (Ostrom, 1996). Indeed, federal funding requirements for stakeholder collaboration, such as co-production, have been shown to influence research practices (Arnott et al., 2020). Arnott and colleagues found that changes in U.S. National Estuarine Research Reserve System program design and funding requirements led to higher levels of interaction between researchers and users. The potential to advance adaptation through usable science coupled with decision makers’ input (Bremer & Meisch, 2017; Lemos et al., 2018; Norström et al., 2020) is what motivates U.S. federally funded climate research programs like NOAA’s CAP to promote the use of co-production (McNie, 2008). The actual practice of co-production, however, can differ widely (Chambers et al., 2021; Mach et al., 2020). Few systematic assessments have tracked the outcomes of co-production (Jagannathan et al., 2020), but its potential impacts are theorized to be broad, ranging from positive consequences such as increasing knowledge and societal transformation (Chambers et al., 2021; Wyborn et al., 2019) to more negative ones, like lack of accountability and the aforementioned reinforcement of inequality (Steen et al., 2018).

1.2. Federal Programs That Support Co-Production

There are a variety of approaches to co-production, each uniquely shaped by the institutions that fund, support, and facilitate these processes (Arnott et al., 2020; Wyborn et al., 2019). In order to assess the range of ways in which equitable co-production can be conceived, the focus of this study is projects funded by federal programs that provide decision-relevant climate change science across the U.S. The programs vary in institutional mission, overall budget, partners or potential users of information, length of existence, and other characteristics. However, each of these programs—the NOAA CAP, Department of the Interior U.S. Geological Survey (DOI USGS) CASCs, and USDA Climate Hubs—promote the practice of co-production in their program’s funding priorities. We briefly describe their approaches below. Each of the case studies described in Box 1 serves as an example of the type of co-production projects the programs fund.

Box 1. The following case studies illustrate funded co-production projects at three federally funded regional climate science centers. They also served as the basis for the study.

USDA SOUTHEAST CLIMATE HUB
Hurricane Preparation and Recovery in the Southeastern United States: Pine Forest Landowners Guide.

The USDA Southeast Climate Hub developed 23 commodity guides to help producers (e.g., farmers, ranchers, and forest land managers) prepare for and recover from hurricanes, a growing threat to the Southeast (McNulty & Gavazzi, 2021). A guide for pine forest landowners (Barlow et al., 2021) in part serves African American audiences who have long been underserved by government (Bailey et al., 2019) and experienced discrimination, contributing to the widespread loss of their land in the last century (Goyke & Dwivedi, 2021). While other minority groups also own forested lands, African American families have experienced the greatest historical declines (Butler et al., 2020). University agricultural extension and USDA Forest Service experts served as the guide’s lead authors (Barlow et al., 2021).

NOAA CAP CONSORTIUM FOR CLIMATE RISK IN THE URBAN NORTHEAST (CCRUN)
Enabling Urban Residents to Adapt to Coastal Flooding: Evidence from New York City Neighborhoods.

Eastern communities along the Rockaway peninsula are both low-income and particularly vulnerable to sea level rise (Vick, 2020). The project was designed as “a co-production process, in which scientists, educators, community leaders, and residents in coastal NYC will collaborate through a series of workshops to understand current and future flood risks and benefits and costs of adaptation actions, which are specific to the communities and occur on time frames relevant to decision making” (Columbia Climate School, 2023). More than a quarter of the population in the Rockaways are immigrants, mostly from Latin American countries (Office of the State Deputy Comptroller for the City of New York, 2022). In communities such as Arverne, Edgemere, Far Rockaway, Hammels and Seaside, more than 53% identify as either Black or Hispanic. The co-production process included community groups who mainly serve the Black and Hispanic communities. An economist at the Columbia Climate School, the university’s school of transdisciplinary climate research, led the project. Partners included a physical oceanographer, a community organization (Rockaway Initiative for Sustainability and Equity), and educational staff from the New York Hall of Science.

USGS ALASKA CLIMATE ADAPTATION SCIENCE CENTER (AK CASC)
Building Resilience Today (BRT).

Loss of permafrost and sea ice, coastal erosion, inland drying, and wildfire threaten the lands and wellbeing of Indigenous people in Alaska (Cochran et al., 2014). The Looking Forward, Looking Back: Building Resilience Today project entailed two trainings and location-based community workshops with five rural communities to provide the information needed to craft climate adaptation plans (Community of Kotlik et al., 2020). The project was led by CASC staff from both USGS and the University of Alaska Fairbanks: a Tribal liaison, natural and social scientists, and project coordinators. They partnered with the Aleutian Pribilof Islands Association and five Alaska Native Communities.

The NOAA CAP program.

CAP’s body of work dates back to the mid-1990s under its former name, the NOAA Regional Integrated Sciences and Assessments (RISA) program (Meadow, 2017). The program operates as a network of regional teams of adaptation experts based at universities and non-profits across the U.S. They partner with decision makers to co-produce knowledge and capacity to advance adaptation, and they tackle complex socioeconomic and climatic stressors (About the Regional Integrated Sciences and Assessments Program, 2023; Combest-Friedman et al., 2019; Parris et al., 2016).

DOI CASCs.

The CASCs were established in 2009 by a Department of the Interior Secretarial Order, tasked with “the development of science and tools to help managers address the effects of climate change on the Nation’s land, water, fish, wildlife, and cultural heritage resources” (U.S. Geological Survey Climate Adaptation Science Centers, 2023). They partner with DOI bureaus, state fish and wildlife agencies, and Native American tribes/Alaska Native Villages (Pfaeffle et al., 2022).

USDA Climate Hubs.

Launched in 2014, the USDA Climate Hubs represent a cross-agency program led by the Agricultural Research Service, Forest Service, and Natural Resources Conservation Service. They develop and deliver information and technologies to support agricultural, forest, and natural resource managers and their associated communities (Steele & Hatfield, 2018; USDA Climate Hubs, 2021).

1.3. Climate Change, Co-Production, and Equity

The three federal programs described above primarily focus on providing decision-relevant science for climate change adaptation. Climate research has been one of the areas in which the study (Bremer & Meisch, 2017) and practice (David-Chavez & Gavin, 2018; Jagannathan et al., 2020) of co-production have grown rapidly. In this “post-normal” science, both uncertainties and decision stakes are high, requiring the involvement of “extended peer communities” (Funtowicz & Ravetz, 1993). The manifold nature of collective action for climate change—creating knowledge or information, building connections/networks, amassing influence, and implementing responses (Skocpol, 2013)—requires stakeholder/rightsholder cross-sectoral, and transdisciplinary efforts, including expertise from a broad range of practitioners and publics (Howarth et al., 2022). Given the diversity of actors, climate risks, and governance structures across the United States, solutions often need to be place-based (Homsy & Warner, 2013), yet at the local level, resources serve as a crucial constraint (Cleveland et al., 2019). Federally funded regional climate science programs identify best practices, facilitate the transfer of knowledge, and leverage resources (Averyt et al., 2018; NOAA, USDA, DOI, 2022).

As governments in the U.S. turn to scientific research to help make policy decisions about climate adaptation, they confront a longstanding conundrum. The ways in which research is conducted for the purpose of societal impact itself can contribute to perceived inequity and potentially perpetuate long-standing inequalities (Rosentraub & Sharp, 1981). The long-term interactions between researchers and stakeholders/rightsholders that increase knowledge usefulness and usability place significant time and other resource demands on participants. These requirements can privilege the involvement of people and organizations with higher levels of income, education, and social capital, giving them greater influence. Exclusion can also result from any number of other factors, such as limited access to information, lack of recognition of a group’s experiences and credibility, or marginalization within the process of generating social meaning (Castán Broto et al., 2022). As such, co-production processes may increase imbalances of power through the selective participation of individuals and organizations in decision-making (Turnhout et al., 2020). Systemic inequalities, inequities, and exclusionary practices deepen and persist without the consideration of social dynamics, histories, and power relations (Amorim-Maia et al., 2022).

The need for climate adaptation is particularly acute among low-income, marginalized, and other socially vulnerable groups (IPCC, 2022). Their higher rates of vulnerability and exposure to climate hazards place them at increased risk that, if unaddressed, exacerbates inequalities. Yet often adaptation policies do not consider equity (Araos et al., 2021). In North America, equity was not addressed in 62% of adaptation planning responses published in the academic literature. Participatory processes are a key recommended component of equitable adaptation (Pelling & Garschagen, 2019), however—as in co-production—the voices of those most marginalized are often those least likely to be heard (Schipper, 2020).

Because of the difficulty in defining the constituent components of co-production processes, not to mention their evaluation (Norström et al., 2020), little research has assessed how to design them in ways that increase not just information access, but also perceived procedural fairness, inclusion, and influence in decision-making by people who are likely to be underrepresented in these activities. Further, the nature of what equitable co-production processes entail is likely to differ depending on who is involved, whether Indigenous Peoples of the Arctic (Yua et al., 2022), urban underserved communities (Perry & Atherton, 2017), or Southern rural populations (Homsy & Warner, 2013). Studies of participatory decision-making have found that people differ in their views of what constitutes a “good” process (Webler et al., 2001).

1.4. Study Goals

We hypothesized that co-production participants—research scientists, boundary organization staff, and people from Native American tribes/Alaska Native Villages or non-Indigenous communities engaged in the processes—would differentially define the components necessary for equitable co-production as it is inherently a subjective question. Boundary organizations specifically work at the interface of science and politics, accountable to both communities in providing them with resources (Guston, 2001). By using a methodological technique developed to study human subjectivity, Q methodology (Stephenson, 1953), we sought to identify areas of consensus, as well as conflict, among these perspectives. Are there some components of equitable co-production processes that appear to be universal, regardless of context?

The project focused on co-production within three projects funded by three U.S. federal programs: the USGS Alaska CASC, NOAA CAP Consortium for Climate Risk in the Urban Northeast, and USDA Southeast Climate Hub. Projects were selected by eliciting regional program directors for projects that focused on co-production and equity, especially projects that aimed to serve people who are underrepresented in science or underserved by federal climate programs. The selected projects span diverse geographies, climate impacts, and communities. An advisory board of federal program managers and representatives of civil society and science organizations provided oversight and participated in the project (Box S1 in Supporting Information S1).

2. Methods

2.1. Q Methodology

Q methodology can be used to evaluate differences in viewpoints between small sub-groups of individuals using inverted factor or principal components analysis (Stephenson, 1953). From a series of diverse statements reflecting how people think about a topic, study participants rank statements closer or further from their own perspective (Stephenson, 1965). The rank-ordered statements allow for the identification of groupings of similar perspectives, indicative of shared narratives. Q methodology has been used extensively within environmental research to explore differences in social perspectives (Webler et al., 2009), including what makes for good communication with stakeholders (Johnson & Chess, 2006).

2.2. Research Steps

The research was conducted in a series of four stages between July 2021 and May 2022 (Figure 1). Methods for each of the steps are described below. All stages of the study were approved by George Mason University’s Institutional Review Board (project #1801755) under a reliance agreement with University of Alaska Fairbanks.

Figure 1.

Figure 1.

Open in a new tab

The research process consisted of a series of four stages.

Stage 1: Interviews with climate science center administrators.

The three climate science centers were chosen for diversity in geography, stakeholders, and relevant climate science and impacts. In fall 2021, co-authors Akerlof, Chase, and Timm conducted structured interviews with the directors or administrators of each of the climate centers. From a list of projects funded between 2016 and 2020, they were asked to identify those that took a co-production approach to research and that addressed issues of relevance to stakeholders, or shared governance partners, who might have reduced access to co-production processes by virtue of race/ethnicity, class, gender, other dimensions of socioeconomic status and their intersection. In order to understand the variability in which co-production projects are conceptualized, co-production was not defined by the research team. From the sub-sample of qualifying projects, the administrators were then asked to select up to three that fit the criteria and that they believed would make good candidates for a case study. They assessed each potential case study using scales adapted from Chambers et al. (2021) to assess the different approaches in terms of purpose, power, politics, and outcome pathways. Lastly, the center directors or administrators described project characteristics and participants, and made a recommendation in selecting the case study (see all study instruments at OSF, https://osf.io/ywf62/).

Stage 2: Interviews with case study project participants.

We pursued a snowball sampling approach in contacting case study project participants, starting with ~60 min structured interviews with the primary investigators. Each was recorded and transcribed. A study by Webler et al. (2001) that assessed perspectives of “good” public participation processes served as a guide in developing the interview protocol. The questions addressed project participation, power, politics, transformation, and defining success. Elements of power and politics are often ignored in co-production processes but can be influential in determining outcomes (Turnhout et al., 2020).

In total, 18 interviews were conducted between November 2021 and April 2022 across the three case studies (Box 1; USDA case study, n = 5; NOAA, n = 7; USGS, n = 6). We interviewed all primary investigators and almost all other identified members of the project team, whether scientists, program managers/evaluators, boundary organization staff, or education and outreach specialists. We also tried to speak with people from the groups that were identified as the focal audience for the project. While this proved to be difficult, in each case we interviewed one participant who was representative of those audiences for whom the initiatives had been designed. Contacting case study participants raised concerns about placing additional burdens on them outside of the scope of the original projects in which they had been engaged. Further, research burdens experienced by these people were often already high. In recognition of the time demand posed by participating in this study, the principal investigator of the research team offered compensation to participate in the interviews and survey ($50 gift cards for each), and also travel and per diem to attend the workshop.

The researchers generated a list of statements on co-production from the interviews and developed a series of codes that reflected the structure of the data. Two members independently coded all of the interview statements and resolved any discrepancies (codebook, Table S1 in Supporting Information S1). This list served as the basis for a final Q set of statements, representing the concourse of viewpoints and opinions expressed during the interviews across each of the categories (Watts & Stenner, 2012). The advisory board (Box S1 in Supporting Information S1) then reviewed statements to ensure they were balanced.

Stage 3: Online survey.

Using a Qualtrics online survey platform employing JavaScript programming developed specifically for Q methodology (Boxes S2–S4 in Supporting Information S1), the interview participants, advisory board members, and other invitees to the workshop with experience in co-production were presented with the statements and asked, “ideally, what should equitable co-production processes and outcomes look like?” They placed each of the statements into nine labeled categories, ranging from “most different from my view” to “most similar to my view.” In performing inverted factor or principal components analysis on the resulting participant rankings, or Q-sorts, the participants become the variables, and their statements the population (Watts & Stenner, 2012). Based on which individuals similarly rank statements, we can identify shared perspectives among groups (Zabala, 2014). These shared discourses—and similarities and differences between them—provided the foundation for the last step of the project. Within the online survey, we also asked respondents to define co-production and equity in their own words (Table S2 in Supporting Information S1).

Thirty-four people completed Q-sorts on what equitable co-production entails prior to the May 2022 workshop (Table 1). However, two people only used a subset of the full ranking scale. They were removed from the data set for the purposes of conducting the principal components analysis. (Their rankings were later correlated with the perspectives to determine the closest to their viewpoint.) Of the remaining 32 respondents, eight were from the case studies. Those remaining were invitees to the May 2022 workshop: local and federal government program managers, academics, and representatives and volunteers of non-governmental organizations.

Table 1.

Respondent Professional Roles by Perspective

Perspective n Case study participants (Federal program) Academia Non-governmental organization Federal/local government
Ways of Knowing and Power (P1) 10 3 (NOAA, USGS) 3 5 2
Participants and Interactions (P2) 11 2 (NOAA, USDA) 1 8 2
Science as Capacity Building (P3) 11 3 (USGS, NOAA) 3 4 4
Totals 32 8 7 17 8
Open in a new tab

Stage 4: Workshop.

The project culminated in a two-day hybrid workshop held in Arlington, Virginia from 12–13 May 2022 (agenda, Table S3 in Supporting Information S1). Case study project participants and advisory board members—as well as other co-production researchers, federal representatives, community members, and civic organization leaders—were invited to attend to draft a framework describing principles for equitable production. Using the Q-sort exercise as a means of identifying which participants held similar perspectives, workshop participants were assigned to break-out groups to discuss whether the findings reflected their views and to develop consensus statements within the group about what equitable co-production should entail (Table S4 in Supporting Information S1). Workshop participants who did not conduct the ranking exercise self-selected into the group closest to their views. This paper represents one of the follow-on activities from the workshop; all participants were invited to join as co-authors.

3. Results

Analysis of the data from the interviews, web-based questionnaire, and workshop follows, providing insights into the perspectives and dimensions of equitable co-production.

3.1. Case Studies

We identified potential case studies with co-production and equity dimensions among the list of funded projects from each of the regional climate science centers. From among the initial nine identified by the interviews with directors and administrators, one case study in each region was selected based on the strength with which they aligned with the criteria (see descriptions, Boxes S5–S7, Table S5 in Supporting Information S1). The final case study selections are described in Box 1. The case studies were initiated and led by researchers or government program managers, which is normative for federally funded projects. However, it is important to note that co-production can be inclusive of projects that are initiated by a wide range of individuals and organizations (Wyborn et al., 2019). The regional climate center administrators described the project approaches using dimensions of co-production developed by Chambers et al. (2021). The project orientations varied considerably in regards to changing the behaviors of individuals, shifting goals and paradigms at higher governance scales, influencing powerful actors, and empowering marginalized actors (Table S5 in Supporting Information S1).

3.2. Interview Results: Five Dimensions of Equitable Co-Production

We abstracted a list of dimensions about equitable co-production from the coded interview transcript statements. The criteria of exhaustiveness and mutual exclusiveness (Neuendorf, 2002) guided the identification of five categories (Table S1 in Supporting Information S1):

  • Outcomes (OC): What are the project outcomes, both intended and achieved?

  • Power (PWR): Which people and forms of knowledge have the power to influence the project across different stages, including preparatory work and planning?

  • Audiences and participation (AUD): Who does or doesn’t participate and how, or reasons they choose not to; audiences for information; access to information, decision-making spaces, and resources to participate.

  • Place-based, community rights and respect (PBD): Place-based focus; what community experiences, challenges, expertise, knowledge, and rights are, or are not, recognized.

  • Interactions (INT): Characteristics of the process: language, communication, process and outcome clarity, longer time periods, and interpersonal contact or relationships.

From the aggregated list of interview statements within each category, representative examples, exhaustive of those coded from the interviews, were chosen for the final concourse of statements to be used in the ranking (Figures 2–6). The final concourse is comprised of 50 statements across the five dimensions. It attempts to capture a diversity of viewpoints from the interviewees on what constitutes equitable co-production. As a result, and by design as part of the Q-methodology technique, the statements sometimes express conflicting viewpoints, for example: “For purposes of scientific rigor, the researcher needs to have ultimate project responsibility” versus “Community members should be in the driver’s seat, including in creating project goals and outcomes.”

Figure 2. Dimension 1—Outcomes (OC).

Figure 2.

Open in a new tab

Outcomes of equitable co-production processes are envisioned as quite broad. The 11 statements that compose this dimension span from education to systemic societal change. Q-sort rankings: −4, “most different from my view” to 4, “most similar to my view.”

Figure 6. Dimension 5—Interactions (INT).

Figure 6.

Open in a new tab

The 10 statements that compose this dimension include language, communication, process and outcome clarity, longer time periods, and interpersonal contact or relationships. Q-sort rankings: −4, “most different from my view” to 4, “most similar to my view.”

3.3. Q-Sort Survey Results: Three Perspectives of Equitable Co-Production

Based on a principle components analysis of the Q-sorts (i.e., participant rankings), we found three perspectives of equitable co-production that are descriptively titled: Ways of Knowing and Power (P1), Participants and Interactions (P2), and Science as Capacity Building (P3). The analysis was performed using qmethod in R (Zabala, 2014, 2022). We examined up to 7-factor solutions using principal components analysis with varimax rotation. Only in the 2- and 3-factor solutions did all participants cleanly fall into one of the perspectives with none loading negatively. The three-factor solution accounted for more total explained variance than the two-factor solution (42.8% vs. 36.2%). Further, each of the three factors accounted for a roughly equal proportion of the total (Table S6 in Supporting Information S1). Solutions that explain more than 35%–40% of the total variance are considered satisfactory (Watts & Stenner, 2012). Interpretation of the perspectives was conducted both through analysis of the Q-sorts (factor matrices and arrays, Tables S7–S13 in Supporting Information S1) and the discussions of workshop participants. Notably, the perspectives are not aligned with the case studies from which the interview statements were taken. Interviewees from the same case study fell into differing perspectives, as did the community members who participated. Nor is there necessarily a relationship with organizational affiliation; individuals from government, academia, and non-governmental organizations ranked across all three (Table 1).

In describing the results below, first we interpret each of the perspectives from the rankings (Figures 2–6), then we describe how the workshop participants construed them within their group discussions. In referring to the statements, the domain and number is followed by the Q-sort ranking, ranging from −4, “most different from my view” to 4, “most similar to my view.” Hence (OC 1:4) refers to statement 1 within the category “Outcomes” (Figure 2) and is ranked “most similar to my view.” The five domain abbreviations include: Outcomes (OC), Power (PWR), Audiences and participation (AUD), Place-based, community rights and respect (PBD), and Interactions (INT). It should be noted that rankings are not an expression of agreement or disagreement, but how closely they represent the view of the respondent. Thus, a respondent could conceivably agree with a statement identified as −4, “most different from my view”—just less strongly than those they ranked more highly.

The perspectives are described below. When rankings from more than one perspective are provided for a specific statement, they appear in numerical order (P1, P2, and P3). While only the Q-sort values are referenced, the statistical significance of differences in z scores informs the interpretation. Significance levels and z scores are provided in the supplementary materials (S9–S13), as is information on the rankings by each perspective (Boxes S8–S10 in Supporting Information S1). In Q methodology, Q-sort values and z scores provide similar information. As described by Zabala and Pascual (2016, p. 5), “The z-score is a weighted average of the values that the Q-sorts most closely related to the factor give to a statement, and it is continuous. Factor scores are integer values based on z-scores and they are used to reconstruct the Q-sort of a factor, which is then interpreted.”

Ways of Knowing and Power (P1).

This perspective focuses on two dimensions of participation in co-production by communities who are affected by the project. First, the co-production process should respect different knowledge systems and ways of knowing (PBD 35:4). Not only should communities have the right to give—or withhold—consent to any project that would affect them, their lands, or resources (PBD 40:4), but they should be in the driver’s seat in creating project goals and outcomes (PWR 16:3) from the outset (AUD 30:3).

Participants and Interactions (P2).

This perspective emphasizes the participatory, communicative, and interactive dimensions of equitable co-production, while honoring the expertise and experiences of communities (AUD 33:4) and their rights to consent (PBD 40:3). Local groups should be involved (AUD 20:3), and community members provided with the information, resources, and technological tools they need to participate (AUD 24:4) and multiple ways to engage (INT 49:4).

Science as Capacity Building (P3).

Co-production outcomes factor more strongly in this perspective than the others: (a) to help people use science and help make science useful to individuals (OC 7:4), (b) to build connections within and external to communities (OC 11:3), and (c) to empower and build capacity for collective action (OC 8:3). Boundary organizations play a core role as partners on the project team with power over decisions (PWR 18:3), and with community buy-in and participation from the outset (AUD 30:3).

3.4. Q-Sort Survey Results: Differences Between the Three Perspectives Across the Dimensions

Dimension 1—Outcomes (OC).

Outcomes, both intended and achieved, were one of the five dimensions of equitable co-production that arose in the interviews. They can be quite broad: research, products, community/societal change, or building connections between science and society. Overall, outcomes did not emerge as a strong characteristic of equitable co-production. People in only one of the perspectives—Science as Capacity Building—highly ranked outcomes as a key component of equitable co-production (Figure 2). Of the 11 outcome statements, six were ranked more highly by this perspective, distinguishing it from other two perspectives (OC 4–9, Figure 2, Table S13 in Supporting Information S1). At the top of the list, “We should help people use science and help make science useful to individuals” was ranked “most similar to my view” by people in the Science as Capacity Building perspective, but considerably lower by others (OC 7:−1, −2, 4). This general statement was made by one of the case study interviewees, but others were more specific in their desired outcomes. For example, one respondent stated, “I wanted each of our local entities to get exposed to climate change issues that we’re currently facing, and to use that information in our decision making efforts within our community.”

Dimension 2—Power (PWR).

Discussions of equity necessarily implicate power. Which people and forms of knowledge shape the project? There was relatively little disagreement about the importance of power for equitable co-production. Across the three perspectives, there was agreement that the standard model of academic research in which the primary investigator is in charge doesn’t strongly align with the way they think about equitable co-production (Figure 3). As one case study interviewee noted: “all the colleges that want to do studies on people; it’s almost like you’re looking at them under a lens under a microscope.”

Figure 3. Dimension 2—Power (PWR).

Figure 3.

Open in a new tab

Seven statements about power refer to which people and forms of knowledge have the ability to influence the project during all stages. Q-sort rankings: −4, “most different from my view” to 4, “most similar to my view.”

“The researcher needs to have ultimate project responsibility” (PWR 13:−4, −4, −4), and “it is up to the core project team to make decisions” (PWR 17:−4, −3, −4) consistently ranked low among respondents. Those with the Ways of Knowing and Power perspective instead describe more democratic forms of governance and decision-making: “feedback from participants should be used to continuously adapt the project” (PWR 15:3) and “community members should be in the driver’s seat, including in creating project goals and outcomes” (PWR 16:3).

However, there were differences in how strongly people associated equitable co-production with local and Indigenous knowledge systems having equal weight with Western science (PWR 14:2, 2, −1). Members of the Ways of Knowing and Participants and Interactions perspectives rated the statement somewhat more highly than those from Science as Capacity Building. It is important to note, however, that the articulation combines local and Indigenous Knowledge, which may have lowered some rankings than if they had been addressed separately.

Dimension 3—Audiences and participation (AUD).

The “co” in co-production inspires many questions. Who does or does not participate? How do they participate? What resources do they need to access information and decision-making spaces? While all three perspectives agreed that project buy-in and participation are needed from the beginning (AUD 30:3, 3, 3), this dimension of equitable co-production was more salient to members of the Participants and Interactions and Science as Capacity Building perspectives than Ways of Knowing and Power (Figure 4). “You know it’s bringing in community members early and for the whole process,” stated one of the interviewees.

Figure 4. Dimension 3—Audiences and Participation (AUD).

Figure 4.

Open in a new tab

Thirteen statements describe who participates at all project stages and their access to information, decision-making spaces, and resources. Q-sort rankings: −4, “most different from my view” to 4, “most similar to my view.”

Those with the Participants and Interactions perspective ranked statements on this dimension particularly high, with those in Science as Capacity Building not far behind. Both groups agreed that local organizations and/or community groups should be involved (AUD 20:3, 4) and that community members should be provided with the information, resources, and technological tools they need to participate (AUD 24:4, 4). In addition, members of Participants and Interactions indicated that the representativeness of both the core project team (AUD 27:3) and the participants (AUD 21:2) was important to their way of thinking. By way of comparison, all four statements received relatively low rankings from those in the Ways of Knowing and Power perspective (AUD 20:1, 24:1, 27:−3, 21:−2).

Dimension 4—Place-based, community rights and respect (PBD).

This dimension localizes equitable co-production and recognizes people and forms of knowledge outside of research institutions. “Their stories were probably the most profound part of this, and I would assume in most co-production processes that’s what comes out is the experiences and the stories,” said one interviewee. Few ranked statements on recognizing community rights as low, but those in Ways of Knowing and Power particularly demonstrated that this dimension was important to their perspective (Figure 5). They selected three of the statements as closest to their views: respecting different ways of knowing (PBD 35:4), communities’ right to give—or withhold—consent to projects (PBD 40:4), and honoring the experience and expertise of communities (PBD 33:3).

Figure 5. Dimension 4—Place-based, Community Rights and Respect (PBD).

Figure 5.

Open in a new tab

This dimension, consisting of nine statements, refers to having a local focus in conducting co-production and what community experiences, challenges, expertise, knowledge, and rights are, or are not, recognized in the process. Q-sort rankings: −4, “most different from my view” to 4, “most similar to my view.”

While there was general agreement across the perspectives of the importance of this dimension of equitable co-production, there was some exception. Members of both the Participants and Interactions and Science as Capacity Building perspectives ordered statements about being place-based (PBD 32:−3, −3) as lower. Those in Science as Capacity Building also indicated that respecting different knowledge systems and ways of knowing was less in line with their views than other criteria for equitable co-production (PBD 35:−2).

Dimension 5—Interactions (INT).

The ways in which people and groups interact in order to co-produce knowledge can take many different forms and proceed along varying timelines. For example, communication can occur through information delivery or within the process of relationship building. All three perspectives generally agreed that co-production processes should provide many ways for participants to engage and at convenient times for them to do so (INT 49:2, 4, 2). Yet, there was more division on other statements on this topic, the most disagreement among the five dimensions (Figure 6). Two of the largest areas of disagreement were on the merits of relationship building (INT 41:4, −2, 0) and whether co-production needs to be a long-term process (INT 44:3, −4, 3). Arguing in favor of their importance, one respondent said, “This work cannot happen without relationships. And those are things that don’t happen in a single grant, you know it’s very long term.”

3.5. Workshop Results: Definitions of Co-Production

Forty people attended the 2-day hybrid workshop to discuss their perspectives of equitable co-production and implications for research and practice (Table 2). The participants were assigned break-out groups by their ranking of the statements and were relatively evenly divided across the three perspectives (P1 n = 9; P2 n = 11; P3 n = 11), gender identification, and organizational affiliation. In addition, two people who completed the ranking but preferred to use a shorter range of values were assigned to groups that correlated most strongly with their preferences (P1, P2). For participants who did not conduct the ranking exercise, and hence were not assigned a break-out group, they were instructed to choose the perspective closest to their own. A fourth break-out group for people who did not align with the original three perspectives was offered, but none of the participants chose to join. The break-out groups reviewed descriptions of the perspectives (Boxes S4–S6 in Supporting Information S1) and definitions of co-production gathered from the online survey (Tables S14 and S15 in Supporting Information S1).

Table 2.

Workshop Participant Perspectives and Organizational Affiliations (n = 40)

Perspectives n %
Ways of Knowing and Power (P1) 10 25.0
Participants and Interactions (P2) 12 30.0
Science as Capacity Building (P3) 11 27.5
No ranking 7 17.5
Organizational affiliation
 Local/federal government 11 27.5
 Non-governmental organization 19 47.5
 Academia 10 25.0
Identification
 He/Him 15 37.5
 She/Her 25 62.5
Open in a new tab

On the first day of the workshop, the three groups were asked if there was one definition of co-production with which they all agreed. Though not all groups could agree on just one, all of them concurred that co-production is a dynamic, collaborative process. The groups’ conclusions are summarized below.

Ways of Knowing and Power (P1).

This group defined co-production as “collaboratively constructing a narrative based on active and equitable participation, mutual respect, and trust in order to meet the self-identified priorities of a distinct self-defined group or community with a commonality of goals.” The definition speaks to the importance of contextualizing co-production in different ways than these processes may have been previously described or documented. In the course of generating the definition, the group raised three important themes: the plurality of knowledge, mutual respect, and empowerment. The group’s definition of co-production is based on the foundational belief that there is not just one way of knowing, but that many different ways exist. This perspective holds that each form of knowledge is equally valuable. The theme of mutual respect builds on this point. The group discussed how mutual respect takes into consideration listening, being patient, and avoiding extractive practices that benefit the researcher more than the other participants. Each partnership should define what participation looks like so that informed consent is possible. Finally, this group discussed the importance of empowerment: building instead of taking, and focusing on assets and goals rather than deficit-based problem framing. Equitable co-production takes into consideration each of these three principles. Further, it recognizes and accounts for contexts that have been, and often still remain, unjust. It strives for a meaningful partnership where all members are committed to looking at something together, whether a definition, issue, problem, or question from multiple perspectives.

Participants and Interactions (P2).

The group converged on the following definition: “An iterative process where people with varying backgrounds, ideas and perspectives collaborate to address mutual priorities and improve outcomes. The process is built on collective expertise, through the equitable sharing of power and decision-making, and grounded in mutual respect, listening and trust. Participants in the process represent the diversity of perspectives in the group whose priorities are being addressed and others who may be impacted.”

Science as Capacity Building (P3).

This group did not come to a consensus definition but did agree on a number of points. Members noted that co-production is sometimes defined variously as a verb (a process) or a noun (a product), but that it should be a verb. The process should evolve and change in reflection of the people who are involved. The group itself cannot be static—it is important to ask people how they can fit into the process and what role they can play. The dynamism is in listening, learning, and doing together. As a result, the focal problem and outcomes may change throughout the course of the process.

3.6. Workshop Results: Consensus on Processes, Outcomes, and Implementation

We hypothesized that the members of each perspective might change their minds about what constituted the core components of co-production after their internal discussions and hearing from those in different groups. In order to assess to what extent a consensus had developed, and whether there were any shared points of agreement across the perspectives, each of the groups was asked to identify their core consensus statements about what equitable co-production entails, whether from the statements that ranked highest for their group, from the full list of 50, or alternately, those that they wrote themselves. With just one exception, members of each perspective selected some subset of the top 8 statements from their group’s ranking (Table 3; Boxes S8–S10 in Supporting Information S1). There was almost no overlap between the core statements of each group. The only statement in which there was some commonality was “Community members should be provided with the information, resources, and technological tools they need to participate.” People in both the Participants and Interactions and Science as Capacity Building perspectives chose this statement as core to their views of equitable co-production.

Table 3.

During the Workshop Each of the Perspectives Identified Core Consensus Statements

Dimension Number Statement P1 P2 P3
Outcomes 7 We should help people use science and help make science useful to individuals X
8 By empowering communities and building their capacity, we create the conditions for collective action X
Power 15 Participants should be included to continuously adapt the project and inform future collaborationsa X
16 Community members should be in the driver’s seat, including in creating project goals and outcomes X
Audiences and Participation 20 Local stakeholder organizations and/or community groups should be involved X
24 Community members should be provided with the information, resources, and technological tools they need to participate X X
27 The core project team should be diverse in representation X
30 Community driven leadership is needed from the beginning of the project’s developmenta X
Place-based, Community Rights and Respect 33 Honoring the expertise and experiences of communities is a critical component X
35 It is all about respecting different knowledge systems and ways of knowing X
39 Community members should be recognized as the experts in their own spaces X
40 Communities have the right to give—or withhold—consent to any project that would affect them, their lands, or resources X
Open in a new tab
a

Statements edited by the group.

The second day of the workshop, the groups were asked three questions for how to implement these ideas: What needs to happen to make this perspective possible? From my position, what do I need to make this perspective possible? From my position, what can I give to make this perspective possible? The workshop participants identified both barriers to conducting equitable co-production and avenues to strengthen the practice (Table 4). See Supplementary Information Part B for more details on both the core statements and implementation.

Table 4.

Opportunities to Improve the Implementation of Equitable Co-Production Processes

Perspective Recommendations
Ways of Knowing and Power (P1)

  • Adopt community-based participatory grant-making approaches

  • Create peer networks and exchanges to disseminate successful approaches and build relationships

  • Establish shared language and meanings in order to make relevant information accessible
Participants and Interactions (P2)

  • Promote innovations in resourcing and governance to support inclusive processes and long-term relationship-building

  • From the start of projects, engage in communication, establish intent, and promote trust

  • Encourage multi-, inter-, and transdisciplinary team research, including the social and behavioral sciences
Science as Capacity Building (P3)

  • Grow equitable co-production partnerships

  • Include community members/representatives on review panels
Open in a new tab

4. Discussion

We demonstrate that there are at least three different perspectives regarding what equitable co-production entails: Ways of Knowing and Power; Participants and Interactions; and Science as Capacity Building. These viewpoints differentially weight the importance of elements associated with five dimensions of co-production practice: outcomes; power; place-based, community rights and respect; audiences and participation; and interactions. Notably, with the exception of “place-based, community rights and respect,” the categories largely align with those from a National Academies report on participation in environmental decision-making: who is involved, when, intensity of involvement, power over the process, and goals (Stern & Dietz, 2008).

Strikingly, even though this research was conducted with federal climate science case studies and more than half of the workshop participants were associated with either government or academia, only one of the perspectives explicitly mentions science within the core components of equitable co-production. Similarly notable are the distinctions in narratives of what equitable co-production entails (Figures 2–6, Table 3), even after exposure to all three perspectives during workshop discussions.

  • Ways of Knowing and Power places an emphasis on democratizing the process of equitable co-production, flipping traditional power dynamics in which academic and government managers lead decision-making. Similarly, it challenges the hierarchy of Western scientific knowledge over other ways of knowing, especially Indigenous Knowledge.

  • Participants and Interactions takes a deliberative approach to equitable co-production in which representativeness aligns with its democratic function. In this perspective, community members’ experiences and expertise should be respected. Further, there is a recognition that they may need information, resources, and technological tools to participate meaningfully in co-production.

  • Science as Capacity Building places science-informed knowledge at the core of societal capacity for collective action. Instead of the models of direct representation described above, community organizations serve as primary partners. Their role is to make decisions in the interests of their members, who may need assistance to effectively participate.

While each of the perspectives describes co-production as highly dynamic and variable in character, the implications for implementing these models differ enormously (Table 4). For example, members of the Ways of Knowing and Power perspective suggest changing funding models to direct grants to communities who decide for themselves what should be funded (Arctic Indigenous Fund, 2023). People in the Participants and Interactions perspective point to paying attention to the governance of co-production, the process of engagement, and team structure. And those in Science as Capacity Building recommend building relationships with community organizations to become better partners in federal programs and peer review processes. All of which begs the question, in which circumstances are specific perspectives, and related co-productive practices, more applicable?

While this research cannot explicitly answer that question, the narratives clearly resemble some contexts and approaches to building knowledge better than others. The Ways of Knowing and Power perspective more closely aligns with decolonizing methodologies (Smith, 2021; Wright et al., 2019) and partnerships with members of Native American tribes/Alaska Native Villages that maintain Indigenous Knowledge systems and have established legal rights to their lands with defined governance boundaries. Alternately, Science as Capacity Building speaks to institutions in which a central mission is conducting and applying research through the lens of Western science. Participants and Interactions addresses the vast space in the middle.

The diversity of perspectives may not be surprising given that co-production has historically arisen from groups with a wide range of interpretations (Bremer & Meisch, 2017; Chambers et al., 2021). For example, emerging literature describe how co-production practices must acknowledge the knowledge sovereignty of Indigenous peoples, instead of seeking to integrate their scholarship within Western scientific models (Latulippe & Klenk, 2020) These are just some of the ethical dimensions of co-production to consider beyond the federal requirements for human subjects research (Wilmer et al., 2021).

Recognizing the potential for differences in perspectives among the vast diversity of humankind serves as a critical first step. “When researchers and stakeholders openly acknowledge differences in professional practices, expectations, and rewards, they establish a foundation for trust and increase the chances of successful collaboration,” observed Wall et al. (2017, p. 551). Intentional conversations conducted at the start of co-production can foster these processes.

5. Implications for Theory and Practice

The relatively low salience of science within the first two perspectives, combined with the community empowerment and capacity building orientation of the third, illustrate the extent to which scientific research took a back seat within these discussions of equitable co-production processes. As such, the discourse of equitable co-production in this study aligns with the 1970s public administration literature on the co-productive role of the public in the provision of public services, enhanced in this case by applied science and technological support. When Elinor Ostrom began writing about co-production of policing she noted:

For some time, most social scientists have conceptualized public agencies producing human services (police, education, welfare) as the primary producers of these services. This conception relegates the citizen to a passive role. Either citizens accept what social service agencies produce as the best they can get, or they direct their participation toward asking agencies to produce better services. …Viewing citizens as coproducers places them in an active role. Citizens are seen either as positively or negatively contributing toward the accomplishment of particular social services

(Ostrom, 1978, p. 102).

Government program officers operate within a system of accountability for providing public services (Bovens et al., 2014), including those with a scientific component. As one example of the ways that federal agencies are trying new models to democratize this process, the NOAA CAP program has recently been experimenting with providing resources to university applicants to design competitions for the distribution of mini-grants to communities and community organizations so that these local organizations can more effectively participate in these co-production projects (NOAA, 2021). But what does it mean for academic scientists to be responsible for providing public services in co-production projects, especially to underserved populations who have long been subject to systemic discrimination? When there are conflicts between the goals of equity, diversity, and democracy, how should trade-offs, such as between direct public participation and indirect representation (Fainstein, 2014), be managed? These questions align with those of active areas of political science, public administration, and urban planning research on the role of public participation in democratic governance (Einstein et al., 2019; Fainstein, 2010). In turn, this literature may help inform the ways in which the scientific community conceptualizes and negotiates these tensions in co-production. Co-production has been termed a “utopian” endeavor within academia, not only challenging power structures in society generally but requiring research institutions to rethink what they do, who does it, and how they operate (Bell & Pahl, 2018).

6. Study Limitations and Future Research

While Q methodology is designed to explore differences in perspectives among small groups, the technique is inherently limited in its ability to generalize to larger populations. Further study is needed to explore whether other groups across different geographies and sociocultural contexts that engage in co-production similarly display these three perspectives, and if so, under which conditions each arises. In addition, each of these perspectives relates to specific practices and associated research methodologies that likely have different outcomes. Follow-on research might address which approaches to co-production result in changes that could be described as “more equitable,” and the extent to which participants are aware of practices such as decolonizing research methodologies. That being said, it is important to acknowledge that what people perceive as equitable and just is related not just to their cognitive evaluations of procedure and distributive outcomes, but their affective responses to their lived experiences (Van den Bos, 2003), which in turn are shaped by their socio-cultural contexts (Mesquita & Leu, 2007).

7. Conclusion

Without any clear consensus on the necessary components of equitable co-production that align across differing perspectives, there can be few assumptions about how others may view them when starting to build new partnerships. Contemplating the varying ways that co-production may be perceived may bring some measure of clarity. Understanding potential perspectives can assist participants in co-production efforts in communicating with each other and considering rules of engagement. Within this capacity for imagination and flexibility lies perhaps the greatest potential for success.

Supplementary Material

Supplement1

Supporting Information may be found in the online version of this article.

Key Points:

  • We identified three perspectives regarding what equitable co-production entails: Ways of Knowing and Power; Participants and Interactions; and Science as Capacity Building

  • The viewpoints are associated with five dimensions: outcomes; power; place-based, community rights and respect; audiences and participation; and interactions

  • Identifying what others may perceive as necessary components for equitable co-production serves as a valuable first step to collaboration

Acknowledgments

Thank you to our project advisory board, workshop participants, and the case study teams for generously sharing their time and expertise. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government. This work was funded by the National Science Foundation (award # 2135538).

Footnotes

Conflict of Interest

The authors declare no conflicts of interest relevant to this study.

Data Availability Statement

The survey ranking and open-ended data, along with the survey and interview protocols, are available at OSF at https://osf.io/ywf62/.

References

  1. About the Regional Integrated Sciences and Assessments Program. (2023). NOAA Climate Program Office. Retrieved from https://cpo.noaa.gov/Divisions-Programs/Climate-and-Societal-Interactions/RISA/About-RISA#739056-about-risa
  2. Amorim-Maia AT, Anguelovski I, Chu E, & Connolly J (2022). Intersectional climate justice: A conceptual pathway for bridging adaptation planning, transformative action, and social equity. Urban Climate, 41, 101053. 10.1016/j.uclim.2021.101053 [DOI] [Google Scholar]
  3. Araos M, Jagannathan K, Shukla R, Ajibade I, Coughlan de Perez E, Davis K, et al. (2021). Equity in human adaptation-related responses: A systematic global review. One Earth, 4(10), 1454–1467. 10.1016/j.oneear.2021.09.001 [DOI] [Google Scholar]
  4. Arctic Indigenous Fund. (2023). Activities. Arctic Indigenous Fund. Retrieved from https://www.arcticindigenousfund.com/projects [Google Scholar]
  5. Arnott JC, Neuenfeldt RJ, & Lemos MC (2020). Co-producing science for sustainability: Can funding change knowledge use? Global Environmental Change, 60, 101979. 10.1016/j.gloenvcha.2019.101979 [DOI] [Google Scholar]
  6. Averyt K, Derner JD, Dilling L, Guerrero R, Joyce L, McNeeley S, et al. (2018). Regional climate response collaboratives: Multi-institutional support for climate resilience. Bulletin of the American Meteorological Society, 99(5), 891–898. 10.1175/BAMS-D-17-0183.1 [DOI] [Google Scholar]
  7. Bailey C, Barlow B, & Dyer J (2019). Practical constraints to timber management among African American owners of heir property. Landscape and Urban Planning, 188, 180–187. 10.1016/j.landurbplan.2019.03.008 [DOI] [Google Scholar]
  8. Barlow B, Andreu M, Asaro C, Maggard A, & Auel J (2021). Hurricane preparation and recovery in the Southeastern United States: Pine forest landowners guide (SRS-GTR-260). U.S. Department of Agriculture, Forest Service, Southern Research Station. Retrieved from https://www.climatehubs.usda.gov/sites/default/files/Pine%20Forest%20Landowners%20Guide%20SE_0.pdf [Google Scholar]
  9. Bell DM, & Pahl K (2018). Co-production: Towards a utopian approach. International Journal of Social Research Methodology, 21(1), 105–117. 10.1080/13645579.2017.1348581 [DOI] [Google Scholar]
  10. Bovens M, Goodin RE, & Schillemans T (2014). The Oxford handbook of public accountability. OUP. [Google Scholar]
  11. Bremer S, & Meisch S (2017). Co-production in climate change research: Reviewing different perspectives. WIREs Climate Change, 8(6), e482. 10.1002/wcc.482 [DOI] [Google Scholar]
  12. Butler SM, Schelhas J, & Butler BJ (2020). Minority family forest owners in the United States. Journal of Forestry, 118(1), 70–85. 10.1093/jofore/fvz060 [DOI] [Google Scholar]
  13. Castán Broto V, Ortiz C, Lipietz B, Osuteye E, Johnson C, Kombe W, et al. (2022). Co-production outcomes for urban equality: Learning from different trajectories of citizens’ involvement in urban change. Current Research in Environmental Sustainability, 4, 100179. 10.1016/j.crsust.2022.100179 [DOI] [Google Scholar]
  14. Chambers JM, Wyborn C, Ryan ME, Reid RS, Riechers M, Serban A, et al. (2021). Six modes of co-production for sustainability. Nature Sustainability, 4(11), 11–996. 10.1038/s41893-021-00755-x [DOI] [Google Scholar]
  15. Clark F, & Illman DL (2001). Dimensions of civic science: Introductory essay. Science Communication, 23(1), 5–27. 10.1177/1075547001023001002 [DOI] [Google Scholar]
  16. Cleveland J, Crowe J, DeBacker L, Munk T, & Plastrik P (2019). Hunting for money: US cities need a system for financing climate resilience and adaptation. Federal Reserve Bank of San Francisco’s Community Development Innovation Review, 14(1), 71–78. [Google Scholar]
  17. Cochran P, Huntington OH, Pungowiyi C, Tom S, Chapin FS, Huntington HP, et al. (2014). Indigenous frameworks for observing and responding to climate change in Alaska. In Maldonado JK, Colombi B, & Pandya R (Eds.), Climate change and Indigenous peoples in the United States: Impacts, experiences and actions (pp. 49–59). Springer International Publishing. 10.1007/978-3-319-05266-3_5 [DOI] [Google Scholar]
  18. Columbia Climate School. (2023). Enabling urban residents to adapt to coastal flooding: Evidence from New York City neighborhoods. Columbia Climate School: Climate, Earth, and Society; [Research Projects]. Retrieved from https://people.climate.columbia.edu/projects/view/1993 [Google Scholar]
  19. Combest-Friedman C, Nierenberg C, & Simpson C (2019). Building a learning network: Reflections from the RISA program. Current Opinion in Environmental Sustainability, 39, 160–166. 10.1016/j.cosust.2019.10.006 [DOI] [Google Scholar]
  20. Community of Kotlik, Littell JS, Fresco N, Toohey RC, & Chase M (Eds.), (2020). Looking forward, looking back: Building resilience today community report (p. 48). Aleutian Pribilof Islands Association. [Google Scholar]
  21. David-Chavez DM, & Gavin MC (2018). A global assessment of Indigenous community engagement in climate research. Environmental Research Letters, 13(12), 123005. 10.1088/1748-9326/aaf300 [DOI] [Google Scholar]
  22. Early W, & Kotzebue AED (2021). More Indigenous knowledge needed to navigate “new Arctic,” scientists say. Alaska Public Media. Retrieved from https://www.alaskapublic.org/2021/02/05/researchers-ask-for-more-indigenous-input-in-national-arctic-science-initiative/ [Google Scholar]
  23. Einstein KL, Glick D, Godinez Puig L, & Palmer M (2022). Still muted: The limited participatory democracy of zoom public meetings. Urban Affairs Review, 10780874211070494. 10.1177/10780874211070494 [DOI] [Google Scholar]
  24. Einstein KL, Glick DM, & Palmer M (2019). Neighborhood defenders: Participatory politics and America’s housing crisis. Cambridge University Press. [Google Scholar]
  25. Fainstein SS (2010). The just city. Cornell University Press. [Google Scholar]
  26. Fainstein SS (2014). The just city. International Journal of Urban Sciences, 18(1), 1–18. 10.1080/12265934.2013.834643 [DOI] [Google Scholar]
  27. Fransman J (2018). Charting a course to an emerging field of “research engagement studies”: A conceptual meta-synthesis. Research for All, 2(2), 185–229. 10.18546/RFA.02.2.02 [DOI] [Google Scholar]
  28. Funtowicz SO, & Ravetz JR (1993). Science for the post-normal age. Futures, 25(7), 739–755. 10.1016/0016-3287(93)90022-L [DOI] [Google Scholar]
  29. Gibbons M (2000). Mode 2 society and the emergence of context-sensitive science. Science and Public Policy, 27(3), 159–163. 10.3152/147154300781782011 [DOI] [Google Scholar]
  30. Gibbons M, Limoges C, Nowotny H, Schwartzman S, Scott P, & Trow M (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. Sage Publications. [Google Scholar]
  31. Goyke N, & Dwivedi P (2021). Does forestry pay? Case studies of four African American family forestland owners in Georgia, United States. Trees, Forests and People, 3, 100047. 10.1016/j.tfp.2020.100047 [DOI] [Google Scholar]
  32. Guston DH (2001). Boundary organizations in environmental policy and science: An introduction. Science, Technology & Human Values, 26(4), 399–408. 10.1177/016224390102600401 [DOI] [Google Scholar]
  33. Hall BL (1992). From margins to center? The development and purpose of participatory research. The American Sociologist, 23(4), 15–28. 10.1007/BF02691928 [DOI] [Google Scholar]
  34. Homsy GC, & Warner ME (2013). Climate change and the co-production of knowledge and policy in rural USA communities. Sociologia Ruralis, 53(3), 291–310. 10.1111/soru.12013 [DOI] [Google Scholar]
  35. Howarth C, Lane M, Morse-Jones S, Brooks K, & Viner D (2022). The ‘co’ in co-production of climate action: Challenging boundaries within and between science, policy and practice. Global Environmental Change, 72, 102445. 10.1016/j.gloenvcha.2021.102445 [DOI] [Google Scholar]
  36. IPCC. (2022). In Pörtner H-O, Roberts D, Tignor M, Poloczanska ES, Mintenbeck K, Alegría A, et al. (Eds.), Climate change 2022: Impacts, adaptation and vulnerability. Working Group II contribution to the sixth assessment report of the Intergovernmental Panel on Climate Change. Cambridge University Press. 10.1017/9781009325844 [DOI] [Google Scholar]
  37. Jagannathan K, Arnott JC, Wyborn C, Klenk N, Mach KJ, Moss RH, & Sjostrom KD (2020). Great expectations? Reconciling the aspiration, outcome, and possibility of co-production. Current Opinion in Environmental Sustainability, 42, 22–29. 10.1016/j.cosust.2019.11.010 [DOI] [Google Scholar]
  38. Jasanoff S (2004). States of knowledge: The co-production of science and social order. Routledge. [Google Scholar]
  39. Johnson BB, & Chess C (2006). From the inside out: Environmental agency views about communications with the public. Risk Analysis, 26(5), 1395–1407. 10.1111/j.1539-6924.2006.00788.x [DOI] [PubMed] [Google Scholar]
  40. Latulippe N, & Klenk N (2020). Making room and moving over: Knowledge co-production, Indigenous knowledge sovereignty and the politics of global environmental change decision-making. Current Opinion in Environmental Sustainability, 42, 7–14. 10.1016/j.cosust.2019.10.010 [DOI] [Google Scholar]
  41. Lemos MC, Arnott JC, Ardoin NM, Baja K, Bednarek AT, Dewulf A, et al. (2018). To co-produce or not to co-produce. Nature Sustainability, 1(12), 722–724. 10.1038/s41893-018-0191-0 [DOI] [Google Scholar]
  42. Lemos MC, & Morehouse BJ (2005). The co-production of science and policy in integrated climate assessments. Global Environmental Change, 15(1), 57–68. 10.1016/j.gloenvcha.2004.09.004 [DOI] [Google Scholar]
  43. Lewin K (1946). Action research and minority problems. Journal of Social Issues, 2(4), 34–46. 10.1111/j.1540-4560.1946.tb02295.x [DOI] [Google Scholar]
  44. Mach KJ, Lemos MC, Meadow AM, Wyborn C, Klenk N, Arnott JC, et al. (2020). Actionable knowledge and the art of engagement. Current Opinion in Environmental Sustainability, 42, 30–37. 10.1016/j.cosust.2020.01.002 [DOI] [Google Scholar]
  45. McNie EC (2008). Co-producing useful climate science for policy: Lessons from the RISA program (PhD Thesis). University of Colorado at Boulder. [Google Scholar]
  46. McNulty S, & Gavazzi M (2021). Hurricane preparation and recovery in the Southeastern United States (SRS-GTR-260). U.S. Department of Agriculture, Forest Service, Southern Research Station. 10.2737/SRS-GTR-260 [DOI] [Google Scholar]
  47. Meadow AM (2017). An ethnohistory of the NOAA RISA program, Climate Assessment for the Southwest. University of Arizona. Retrieved from https://climas.arizona.edu/publication/report/ethnohistory-noaa-risa-program [Google Scholar]
  48. Mesquita B, & Leu J (2007). The cultural psychology of emotion. In Handbook of cultural psychology (pp. 734–759). The Guilford Press. [Google Scholar]
  49. Minkler M (2010). Linking science and policy through community-based participatory research to study and address health disparities. American Journal of Public Health, 100(S1), S81–S87. 10.2105/AJPH.2009.165720 [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. National Science Foundation. (2020). The State of U.S. Science and Engineering 2020. National Center for Science and Engineering Statistics (NCSES). Retrieved from https://ncses.nsf.gov/pubs/nsb20201/u-s-r-d-performance-and-funding#figureCtr886 [Google Scholar]
  51. National Science Foundation. (2022). Research and development: U.S. trends and international comparisons. In The State of U.S. Science and Engineering 2022. National Center for Science and Engineering Statistics (NCSES). Retrieved from https://ncses.nsf.gov/pubs/nsb20225/recent-trends-in-federal-support-for-u-s-r-d#distribution-of-federal-funding-of-r-d-by-performer-and-type-of-r-d [Google Scholar]
  52. NCSES. (2022). Definitions of research and development: An annotated compilation of official sources (NCSES 22–209). National Center for Science and Engineering Statistics, National Science Foundation. Retrieved from https://ncses.nsf.gov/pubs/ncses22209/assets/ncses22209.pdf [Google Scholar]
  53. Neuendorf KA (2002). The content analysis guidebook. Sage Publications. [Google Scholar]
  54. NOAA. (2021). Climate Program Office (CPO), Regional Integrated Science and Assessments (RISA) FY2022 [NOAA-OAR-CPO-2022–2007019]. Oceanic and Atmospheric Research (OAR), National Oceanic and Atmospheric Administration (NOAA), Department of Commerce. Retrieved from https://www.grants.gov/web/grants/view-opportunity.html?oppId=335881 [Google Scholar]
  55. NOAA, USDA, DOI. (2022). Interagency collaboration: Three federal climate networks partner to enhance regional adaptation and resilience. DOI/NOAA/USDA Climate Adaptation Interagency Collaboration. Retrieved from https://toolkit.climate.gov/sites/default/files/CPO_Intera-gencyCollaboration_20220105%20-%20FINAL.pdf [Google Scholar]
  56. Norström AV, Cvitanovic C, Löf MF, West S, Wyborn C, Balvanera P, et al. (2020). Principles for knowledge co-production in sustainability research. Nature Sustainability, 3(3), 182–190. 10.1038/s41893-019-0448-2 [DOI] [Google Scholar]
  57. Office of the State Deputy Comptroller for the City of New York. (2022). An economic and demographic snapshot of the Rockaways (Report 7–2023). Office of the New York State Comptroller. Retrieved from https://www.osc.state.ny.us/files/reports/osdc/pdf/report-7-2023.pdf [Google Scholar]
  58. Ostrom E (1978). Citizen participation and policing: What do we know? Journal of Voluntary Action Research, 7(1–2), 102–108. 10.1177/089976407800700110 [DOI] [Google Scholar]
  59. Ostrom E (1996). Crossing the great divide: Coproduction, synergy, and development. World Development, 24(6), 1073–1087. 10.1016/0305-750X(96)00023-X [DOI] [Google Scholar]
  60. Parris AS, Garfin DGM, Dow PK, Meyer DR, & Close DSL (2016). Climate in context: Science and society partnering for adaptation. John Wiley & Sons. [Google Scholar]
  61. Pelling M, & Garschagen M (2019). Put equity first in climate adaptation. Nature, 569(7756), 327–329. 10.1038/d41586-019-01497-9 [DOI] [PubMed] [Google Scholar]
  62. Perry B, & Atherton M (2017). Beyond critique: The value of co-production in realising just cities? Local Environment, 22(sup1), 36–51. 10.1080/13549839.2017.1297389 [DOI] [Google Scholar]
  63. Pfaeffle T, O’Malley R, Bamzai A, & Tangen S (2022). Training and capacity building activities of Climate Adaptation Science Centers for the benefit of Tribal and Indigenous communities, 2010–2019. US Geological Survey. Retrieved from https://pubs.er.usgs.gov/publication/gip217/full [Google Scholar]
  64. Pulwarty RS, Simpson C, & Nierenberg CR (2009). The Regional Integrated Sciences and Assessments (RISA) Program: Crafting effective assessments for the long haul. Integrated Regional Assessment of Global Climate Change, 367, 393. [Google Scholar]
  65. Rosentraub MS, & Sharp EB (1981). Consumers as producers of social services: Coproduction and the level of social services. Southern Review of Public Administration, 4(4), 502–539. [Google Scholar]
  66. Schipper ELF (2020). Maladaptation: When adaptation to climate change goes very wrong. One Earth, 3(4), 409–414. 10.1016/j.oneear.2020.09.014 [DOI] [Google Scholar]
  67. Skocpol T (2013). Naming the problem: What it will take to counter extremism and engage Americans in the fight against global warming. In Prepared for the Symposium on the Politics of America’s Fight against Global Warming. Harvard University. [Google Scholar]
  68. Smith LT (2021). Decolonizing methodologies: Research and Indigenous Peoples (3rd ed.). Zed Books. [Google Scholar]
  69. Steele R, & Hatfield JL (2018). Navigating climate-related challenges on working lands: A special issue by the USDA Climate Hubs and their partners. Climatic Change, 146(1), 1–3. 10.1007/s10584-017-2129-3 [DOI] [Google Scholar]
  70. Steen T, Brandsen T, & Verschuere B (2018). The dark side of co-creation and co-production: Seven evils. In Brandsen T, Steen T, & Verschuere B (Eds.), Co-production and co-creation: Engaging citizens in public services (pp. 284–293). Taylor and Francis. Retrieved from https://library.oapen.org/handle/20.500.12657/25001 [Google Scholar]
  71. Stephenson W (1953). The study of behavior; Q-technique and its methodology (pp. 9–376). University of Chicago Press. [Google Scholar]
  72. Stephenson W (1965). Perspectives in psychology: XXIII Definition of opinion, attitude and belief. Psychological Record, 15(2), 281–288. 10.1007/BF03393596 [DOI] [Google Scholar]
  73. Stern PC, & Dietz T (2008). Public participation in environmental assessment and decision making. National Academies Press. [Google Scholar]
  74. Turnhout E, Metze T, Wyborn C, Klenk N, & Louder E (2020). The politics of co-production: Participation, power, and transformation. Current Opinion in Environmental Sustainability, 42, 15–21. 10.1016/j.cosust.2019.11.009 [DOI] [Google Scholar]
  75. USDA Climate Hubs. (2021). USDA Climate Hubs strategic plan, 2020–2025. U.S. Department of Agriculture (USDA). Retrieved from https://www.climatehubs.usda.gov/content/usda-climate-hubs-strategic-plan-2020-2025 [Google Scholar]
  76. USGCRP. (2023). Agencies. U.S. Global Change Research Program. Retrieved from https://www.globalchange.gov/agencies [Google Scholar]
  77. U.S. Geological Survey Climate Adaptation Science Centers. (2023). History of the CASCs. USGS. Retrieved from https://www.usgs.gov/programs/climate-adaptation-science-centers/history-cascs [Google Scholar]
  78. Van den Bos K (2003). On the subjective quality of social justice: The role of affect as information in the psychology of justice judgments. Journal of Personality and Social Psychology, 85(3), 482–498. 10.1037/0022-3514.85.3.482 [DOI] [PubMed] [Google Scholar]
  79. Vick R (2020). Thousands of low-income renters in Queens at risk from rising seas, report finds. Queens Daily Eagle. Retrieved from https://queenseagle.com/all/tenants-in-1800-affordable-queens-apartments-at-risk-from-rising-seas-report-finds [Google Scholar]
  80. Wall TU, McNie E, & Garfin GM (2017). Use-inspired science: Making science useable by and useful to decision makers. Frontiers in Ecology and the Environment, 15(10), 551–559. 10.1002/fee.1735 [DOI] [Google Scholar]
  81. Wandersman A (2003). Community science: Bridging the gap between science and practice with community-centered models. American Journal of Community Psychology, 31(3–4), 227–242. 10.1023/A:1023954503247 [DOI] [PubMed] [Google Scholar]
  82. Watts S, & Stenner P (2012). Doing Q methodological research: Theory, method and interpretation. Sage Publications. [Google Scholar]
  83. Webler T, Danielson S, & Tuler S (2009). Using Q method to reveal social perspectives in environmental research. Social and Environmental Research Institute. Retrieved from http://www.seri-us.org/sites/default/files/Qprimer.pdf [Google Scholar]
  84. Webler T, Tuler S, & Krueger R (2001). What is a good public participation process? Five perspectives from the public. Environmental Management, 27(3), 435–450. 10.1007/s002670010160 [DOI] [PubMed] [Google Scholar]
  85. Wilmer H, Meadow AM, Brymer AB, Carroll SR, Ferguson DB, Garba I, et al. (2021). Expanded ethical principles for research partnership and transdisciplinary natural resource management science. Environmental Management, 68(4), 453–467. 10.1007/s00267-021-01508-4 [DOI] [PubMed] [Google Scholar]
  86. Wright AL, Gabel C, Ballantyne M, Jack SM, & Wahoush O (2019). Using two-eyed seeing in research with Indigenous people: An integrative review. International Journal of Qualitative Methods, 18, 1609406919869695. 10.1177/1609406919869695 [DOI] [Google Scholar]
  87. Wyborn C, Datta A, Montana J, Ryan M, Leith P, Chaffin B, et al. (2019). Co-producing sustainability: Reordering the governance of science, policy, and practice. Annual Review of Environment and Resources, 44(1), 319–346. 10.1146/annurev-environ-101718-033103 [DOI] [Google Scholar]
  88. Young S, Mallory B, & McCarthy G (2021). The path to achieving Justice40. Office of Management and Budget, The White House. Retrieved from https://www.whitehouse.gov/omb/briefing-room/2021/07/20/the-path-to-achieving-justice40/ [Google Scholar]
  89. Yua E, Raymond-Yakoubian J, Daniel R, & Behe C (2022). A framework for co-production of knowledge in the context of Arctic research. Ecology and Society, 27(1), art34. 10.5751/ES-12960-270134 [DOI] [Google Scholar]
  90. Zabala A (2014). qmethod: A package to explore human perspectives using Q methodology. The R Journal, 6(2), 163–173. 10.32614/rj-2014-032 [DOI] [Google Scholar]
  91. Zabala A (2022). Qmethod, an R package to analyse Q methodology data. Retrieved from http://aiorazabala.github.io/qmethod/
  92. Zabala A, & Pascual U (2016). Bootstrapping Q methodology to improve the understanding of human perspectives. PLoS One, 11(2), e0148087. 10.1371/journal.pone.0148087 [DOI] [PMC free article] [PubMed] [Google Scholar]

References From the Supporting Information

  1. Adame F (2021). Meaningful collaborations can end ‘helicopter research’. Nature, 10. 10.1038/d41586-021-01795-1 [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement1
NIHMS2131326-supplement-Supplement1.docx (242.6KB, docx)

Data Availability Statement

The survey ranking and open-ended data, along with the survey and interview protocols, are available at OSF at https://osf.io/ywf62/.

RESOURCES