Abstract
Background:
Lower-cost air quality sensors (hundreds to thousands of dollars) are now available to individuals and communities. This technology is undergoing a rapid and fragmented evolution, resulting in sensors that have uncertain data quality, measure different air pollutants and possess a variety of design attributes. Why and how individuals and communities choose to use sensors is arguably influenced by social context. For example, community experiences with environmental exposures and health effects and related interactions with industry and government can affect trust in traditional air quality monitoring. To date, little social science research has been conducted to evaluate why or how sensors, and sensor data, are used by individuals and communities, or how the introduction of sensors changes the relationship between communities and air quality managers.
Objectives:
This commentary uses a risk governance/responsible innovation framework to identify opportunities for interdisciplinary research that brings together social scientists with air quality researchers involved in developing, testing, and deploying sensors in communities.
Discussion:
Potential areas for social science research include communities of sensor users; drivers for use of sensors and sensor data; behavioral, socio-political, and ethical implications of introducing sensors into communities; assessing methods for communicating sensor data; and harnessing crowdsourcing capabilities to analyze sensor data.
Conclusions:
Social sciences can enhance understanding of perceptions, attitudes, behaviors, and other human factors that drive levels of engagement with and trust in different types of air quality data. New transdisciplinary research bridging social sciences, natural sciences, engineering, and design fields of study, and involving citizen scientists working with professionals from a variety of backgrounds, can increase our understanding of air sensor technology use and its impacts on air quality and public health.
Keywords: air pollution, social science, citizen science, monitoring, exposure
1.1. Introduction
Rapid developments in technology are fueling an influx of hand-held, wearable technologies (e.g. smartphones, activity tracking devices, heart monitoring) engineered to collect real-time, localized data about individuals and their immediate surroundings. One increasingly relevant example of these technologies is low-cost, portable air quality sensors. The rapid and decentralized evolution of these technologies has resulted in a variety of affordable sensors with different mechanisms for data collection (Figure 1) that have uncertain data quality, measure different air pollutants, and possess a variety of design attributes, including how readings are presented or interpreted (See Jiao et al, 2017 and Lewis et al, 2016 for recent performance evaluations of low cost sensors). The developers of these technologies are varied, including traditional air measurement technology companies, crowd-funded start-ups, large information technology (IT) corporations, and community organizations. While this technology is proliferating, research to inform the translation of air sensor data into information that might guide an individual’s decisions about daily activities remains limited. Despite these uncertainties, the potential demand for sensor technology is high, driven by widespread concern about air pollution as well as an interest in reducing personal exposure. The adoption of low-cost air quality sensors by both public and private sectors, for a diverse set of applications, portends expansive use and widespread circulation of sensor-based air quality data. Further research is needed to elucidate how air quality sensors and their data are being used and to better understand the groups and individuals who use them.
To fully grasp the potential impact of the ubiquitous availability and public use of air pollution sensor technology it is important to understand 1) how communities of sensor users grow and are sustained, 2) the drivers behind individual and community-level air sensor use and data collection, 3) the behavioral, socio-political, and ethical implications of introducing sensors into communities, 4) how sensor data is communicated within and across communities, and 5) how crowdsourcing capabilities can be harnessed to achieve greater benefits of community sensor networks. Gaining understanding in these areas will require application of social science theories and methods. Currently, there are few studies that evaluate social or economic implications of low-cost air quality sensors (Zappi et al. 2012; Bales et al. 2014; Willett et al. 2010). A recent article reviews the use of community-based participatory research (CBPR) approaches to study air pollution in communities (Commodore et al, 2017). However, the article focuses on communities as researchers, and sensors as objects of research, rather than on the motivations for non-research use of sensors or the social context of introducing sensor technologies into communities. Theories of risk governance (the process by which decisions about risk are determined) and responsible research and innovation provide useful context for considering how the development of air quality sensors and their subsequent introduction into communities can impact individual and group level decisions (van Asselt and Ren, 2011; Owen, Macnaghten and Stilgoe, 2012). There are well established health risks from exposures to air pollution (U.S. EPA, 2009; U.S. EPA, 2013) that can potentially be mitigated through use of air quality sensors, and also potential risks to health and community welfare that can arise from the introduction of sensors into communities, for example if poor quality data leads to behaviors that increase exposure. Government processes that shape policy for new technology determine the potential action pathways for groups (such as non-governmental organizations or community groups) and individuals. Conversely, individuals and groups interactions with new technology, from the initial introduction of an innovation to relegation of obsolete technologies to marginalized communities, often influence the policy process. Vulnerable or overburdened communities that face multiple social or environmental stressors may have different responses to new technologies and the interpretation of expert versus local knowledge (Corburn, 2005). Experts are defined here to be individuals with formal education and professional experience in a relevant scientific field. Citizen scientists are members of the general public (not professional scientists or analysts) who are involved in scientific research and activities (Cohn, 2008; Silvertown, 2009). Citizen scientists and engaged community members with local knowledge often engage in scientific studies or analyses using scientific methods or local experiential knowledge to better understand local environmental conditions and risks (Aoki et al, 2017). Both experts, citizen scientists, and engaged community members contribute to the overall understanding of environmental quality and risks in local communities. Better understanding of the effects of such a knowledge-enabling technology as air quality sensors, particularly when adopted by communities vulnerable to marginalization (communities with low access to resources, opportunities, and agency)can lead to identification of opportunities for collaborative policy solutions and reduced environmental health burdens. The process of engaging in monitoring of local air quality conditions can help to increase environmental literacy in communities and build capacity of communities to engage both in partnering with academic researchers, and in developing local actions to reduce air pollution exposures and improve public health.
In this commentary, we discuss the limited literature on application of the social sciences in the area of air quality sensors. We identify opportunities for interdisciplinary research that brings together concepts and methods from a range of social science disciplines with environmental science, engineering and user interface (UI)/user experience (UX) design communities that are developing, testing, and deploying air quality sensor technologies. We lay out the nature of the challenges associated with sensor data generation, interpretation, and analysis. We identify opportunities for collaboration with communities and organizations to better understand how and why sensors are being used, and how technological innovations may be able to improve the ability of communities and individuals to reduce exposures to air pollution and improve individual and public health.
1.2. Discussion
Multiple social science disciplines have addressed the development and adoption of environmental technologies and have provided theoretical frameworks for the examination of these processes. In this paper we choose to make use of the complementary conceptual frameworks of risk governance (van Asselt and Ren, 2011; Renn and Schweizer) and responsible research and innovation (Owen, Macnaghten and Stilgoe, 2012; Stilgoe, Owen & Macnaghten, 2013).
This is not the only conceptual framework that can guide thinking about how individuals and communities respond to the availability of lower cost sensors. Pritchard and Gabrys (2016) use a framework of “perceptive and affective problematics” to discuss how interactions among individuals, communities and technology co-produce knowledge and mobilize publics towards certain political objectives. This is a useful construct for thinking about the specific question of how relationships between individuals and communities can affect how sensor information is collected and used, both in generating knowledge and in effecting political change. For the purposes of this inquiry, we find the framework of risk governance and responsible research and innovation to be applicable to a wide variety of potential impacts of air quality sensors and thus adopt that framework for our discussion.
The concepts of responsible research and innovation focuses on the democratic, inclusive governance of new sciences and technologies, including, but not limited to, air quality sensors (Owen, Macnaghten and Stilgoe, 2012; Stilgoe, Owen & Macnaghten, 2013). There are four dimensions to the framework: anticipation, reflexivity, inclusion and responsiveness (Stilgoe, Owen & Macnaghten, 2013). Anticipation asks researchers to think of future uses and scenarios for the science or technology they are developing and related risks. Closely related, reflexivity challenges researchers to recognize the limits of their knowledge and to directly reflect on the morality of their work. Inclusion focuses on the substantive input of stakeholders, while striving to identify and dissolve barriers precluding those who may not typically be involved in such innovative processes. Finally, innovation must be able to respond to the challenges provided by increased anticipation, reflexivity and inclusion. Conversely, while responsible innovation focuses on the early, innovative stages of a new technology, inclusive risk governance focuses on the risks posed by established technology. Specifically, risk governance addresses the various ways in which individuals and institutions manage risks that are surrounded by uncertainty, complexity, and ambiguity (van Asselt and Ren, 2011).
The development and deployment of air quality sensor technology is currently undertaken by a diversity of organizations. On one end of the spectrum is the AirCasting Airbeam sensor technology (http://aircasting.org/about) that was developed by a community organization and designed to be open source, as well as streaming all data from users into a common platform to develop a crowd-sourced air quality map. The variations of the design include multiple air pollutant sensors, biometric sensors, and noise data capture from Android phones. Meanwhile, on the other end of the spectrum are emerging sensor networks implemented in major cities by larger organizations (e.g, large corporations, universities, government), with data made available to the public. In the latter case, some larger organizations have conducted information sharing sessions to seek public input on where to locate the technologies, which may help build trust and engagement of the community. An example of this is the Minnesota Pollution Control Agency (MPCA) planning process to implement sensors in a number of locations in Minneapolis / St. Paul (https://www.pca.state.mn.us/air/assessing-urban-air-quality-project). The MPCA held five “open houses” to allow the public to hear about the project and provide input on sensor locations.
The availability of lower-cost air quality sensors may change the nature of risk governance by reducing uncertainty, but can also increase ambiguity and complexity, and add new dynamics related to who “owns” or has rights to knowledge about air quality. The overall impact of these changes on how individuals and institutions interact with each other to understand and manage air pollution risks is ripe for study. Air pollution is a systemic risk, due to ubiquitous sources, mobile and heterogeneous exposed populations, and varying exposures due to meteorology, topography, the build environment, and population activity patterns. However, individuals and institutions are likely to respond to air pollution risks in different ways, “according to their own risk constructs and images.” (van Asselt and Ren, 2011). When there are multiple sources of data available from official government monitors, sensor networks implemented by private or academic institutions, as well as community or individual air quality sensors, ambiguity may result regarding the nature of the risks and the appropriate ways to manage those risks.
1.2.2. Review of the literature on social and behavioral aspects of sensor use
To date, there are few research applications where social scientists have directly considered social implications of introducing air pollution sensors into communities. There are almost no quantitative studies of sensor use by different types of users, and only a few qualitative survey-based assessments of how and why users are employing air quality sensors. The CitiSense project (Zappi et al, 2012) evaluated how 16 participants responded to having instant feedback about the air quality in their vicinity. The project team provided an online mapping tool that allowed the users to view their past and current air quality based on measurements from a sensor node connected to a smart phone. Participants were interviewed to understand their experiences using the sensor device and how the available data influenced their behaviors. Over the course of the study participants had few issues with using the sensors, but voiced concerns regarding data interpretation and knowing how to react to information provided by the sensor. For example, they felt that historical data was needed to more fully interpret current air quality. Participants also wanted more gradations of air quality than provided by the EPA Air Quality Index (AQI) to help distinguish their immediate air quality. Participants utilized social media to share their measurements with co-workers and friends. They also found that access to personalized air quality information was useful, both in revising prior beliefs about air pollution, and learning how their activity patterns and behaviors affected their exposures.
Willett et al (2012) adopted a slightly different approach relying on structured interviews with several dozen community members who were engaged in collecting measurements with personal air quality sensors (measuring carbon monoxide (CO), nitrogen dioxide (NO2), and ozone (O3)). The authors explored how improved design of mobile air quality monitoring systems could enhance the process of knowledge production such as combining information from sensors with individual knowledge of emissions sources and experiences with health impacts. Based on their findings, the study authors developed a system called the Common Sense Community (http://www.communitysensing.org/) to facilitate community analysis of air quality data collected by sensors. Key aspects of the design of the analysis system include a focus on personal exposure or hotspots, use of local data and knowledge (for example locations of emissions sources), and avoiding use of technical jargon and information overload. The authors evaluated their community-based system in an area with known air quality problems, and found that the system was useful to participants to understand local pollution and communicate insights to policy makers and others in their community.
Oltra et al (2017) evaluated changes in perceptions and knowledge of air quality for two small groups of study participants in Barcelona. One group (n=12) received personal NO2 sensors (Aeroqual 505 series handheld monitor with a NO2 NWGSS64 sensor) and collected air quality readings over a one-week period. The other group (n=16) passively received information on air quality and health using videos, press clippings, and information pamphlets. Using focus groups, they found that the participants who used the NO2 sensors had a more quantitative understanding of pollution levels compared to those receiving general information. This gave them a sense of knowing when the air was highly polluted. However, compared to the group that received general air quality information which included information about health effects, the group using NO2 sensors became more focused on the quantitative readings and expressed less concern about health effects, except when readings were very high. Both groups reported low levels of self-efficacy, believing they could do little to avoid the harmful effects of air pollution, and indicated limited willingness to engage in protective behaviors (masks or changing commuting routes) due to concerns about costs, convenience, or esthetics. Recent literature on environmental health literacy indicates that by educating individuals and communities and raising awareness of environmental risks and strategies to reduce those risks, feelings of self-efficacy can be increased (Finn and O’Fallon, 2017). These increases in self-efficacy can in turn lead to improved decision making at the individual and community levels to reduce adverse health outcomes.
The current array of published commentaries (O’Rourke and Macey, 2003; Monahan and Mokos, 2010; McElfish, Pendergrass, and Fox, 2016; Scott and Barnett, 2009; Gabrys and Prichard, 2016) were mostly published prior to the recent influx of low-cost air quality sensors, but addressed some of the potential social implications of citizen-based air monitoring that are highly relevant today. In brief, they conclude that public sampling of environmental contaminants has resulted primarily in increased community awareness of air pollution and responses (including increased political advocacy), with the potential in some cases to impact broader policy actions and change industrial processes (including on-site monitoring). They also identify the potential for sensors to transfer responsibility for risk reduction from the polluters and regulators to the vulnerable populations themselves, potentially exacerbating health disparities and reducing feelings of empowerment in affected areas. In addition, they note the legal and administrative hurdles faced by citizen scientists, including getting data they collect accepted for use in official decision-making about environmental issues, the need for data quality and peer review protocols, considerations of data privacy issues, and rules for introducing evidence into legal proceedings and enforcement actions. The recent article by Commodore et al (2017) notes that CBPR and citizen science efforts related to air quality monitoring (all monitoring, not just using lower-cost sensors) were driven in part by community concerns about disease burdens and living in a “toxic” environment due to proximity to roadways and industrial emissions sources, as well as a desire to know more about air quality in their community and a desire to reduce health burdens due to air pollution exposure. English et al (2017), reporting on a community monitoring effort in Imperial County, California, highlight the importance of engaging community members in all aspects of monitoring air quality when establishing a community wide air monitoring network, noting that the process of engaging community members as citizen scientists builds trust and can address concerns about scientific validity and sustainability of the monitoring network. Community members can serve in multiple roles during a monitoring research effort, helping to identify appropriate monitoring locations, develop effective communications plans, and translate research findings into actions to address sources of air pollution and exposures.
1.2.2. Research opportunities
1.2.2.1. Growing and sustaining physical and virtual communities of sensor users
Given the spatially and temporally dynamic nature of air pollution, there is significant value in combining numerous individual sensor data sets collected by individuals or groups into a larger database informing community-wide air quality. However, sustained participation in the endeavor is essential to the task. Investigative mechanisms that incentivize sustained participation may be helpful to groups planning sensor deployment campaigns. Study participants are generally more willing to participate (and share personal information like health or activity data) when research produces actionable results. In addition, individuals may be more likely to initiate and sustain participation if well-known community organizers or key community members endorse the campaign. (Willett et al, 2012)
The complexity involved in interpreting air pollution trends, along with the potential technical barriers for sensor use, may be a limiting factor for community use of air quality sensors. Understanding potential limiting factors as well as catalysts, such as developing local citizen technical experts, is important to understand how sensor technology may be adopted and used. EPA has supported the advancement of community member “experts” by launching the Community Air Monitoring Training and Air Sensors Toolbox online suite of resources for citizen scientists (https://www.epa.gov/air-research/air-sensor-toolbox-citizen-scientists-resources). Research that improves understanding of the “expert” within the community can be helpful to organizations seeking to employ community-based air quality sensors. Qualitative methods, such as one-on-one interviews or focus groups, can be very useful tools to understand how different communities and individuals view the role of experts within their specific social or cultural context (Scammel, 2010). Such research could help assess whether citizens with greater technical knowledge may be able to serve as points of contact between academic scientists, environmental groups, and communities, and act as liaisons to train and maintain broader groups of citizen scientists. English et al (2017) emphasize that sustaining a community monitoring program after a directly funded research effort has concluded will require training citizens to maintain monitors and interpret and communicate collected data in appropriate and useful ways.
Maximizing knowledge generation through crowdsourced air quality sensor data will require evaluating the most effective ways to organize the community of data producers, analysts, and users. This may require additional research into the nature of power dynamics that determine who collects data, the quality constraints on sensors used for different purposes, as well as who controls the flow, storage, and access to sensor data. Pritchard and Gabrys (2016) note that citizen sensing can challenge the established “truth” provided by traditional monitoring which can lead to tensions between communities and government agencies that have previously controlled air quality knowledge. Community-collected sensor data may be a catalyst for increasing individual and community awareness of air quality and causes of pollution events, and developing strategies to reduce pollution exposure.
Social scientists using a variety of methodological tools and techniques can help elucidate the social context in how sensor data are collected, analyzed, and presented which may result in either collaborative solutions, or conflict, between citizens, polluters, and community governance in addressing air quality problems. For example, a mental models methodology could be utilized. Mental models are ways of explaining how people understand information or knowledge (Gentner and Stevens, 2014). Application of a mental models approach would involve the examination of knowledge and value gaps in the collection, analysis and communication of sensor data between scientists and community partners (Morgan et al, 2002).
1.2.2.2. Understanding drivers behind individual and community-level air sensor use and data collection
Theories of responsible research and innovation address the role of the responsiveness of innovation to meeting the needs of the public, and the responsibility for results of the innovation given uncertain and unpredictable consequences. (Owen, Macnaghten and Stilgoe, 2012). Within this context, understanding of the nature of individual and community experiences with air pollution and the perceived need for additional air quality information can help to evaluate how well existing air quality sensors are meeting the needs for innovation. In addition, evaluations of current uses of air quality sensors, including who actually uses sensors, and how sensor awareness and use relates to underlying demographics and socioeconomic status, can help determine which types of needs are currently being addressed through sensor innovations (Moore, 2014). For example, environmental justice communities may view sensors as a way to document a previously identified (using other indicators such as observed health effects or visible pollution) environmental problem (Sadd et al, 2014). Concerned communities may want to have full ownership over the sensor technology to “ground truth” external air quality information sources (e.g., regulatory air monitoring, state or federal emissions inventories), which may build or erode trust in their relationships with government and academic investigators (Sadd et al, 2014). A potentially useful research question applicable to individuals is “to what extent are sensors employed to affirm predetermined perceptions or goals, e.g., comparing against an air quality benchmark to adjust activities, versus the development of open questions or hypotheses, e.g. if I change my bike route, will I see a change in my exposure to PM?”
It is also important to improve our understanding of how the motivational and confirmation biases of citizen scientists may affect their views of sensor data. Motivational biases arise when self-interest or social pressures distort judgements and decisions (Montibeller and von Winterfeldt, 2015). Confirmation biases occur when individuals selectively interpret new data to confirm a pre-existing belief (Montibeller and von Winterfeld, 2015). Questions that arise related to motivational and confirmation biases about sensor data include how does the strength of prior beliefs impact the ability/willingness to update those beliefs based on sensor data? To what extent do individuals tie “success” in using a sensor to findings of high variation in or high concentrations of air pollution (Willett, 2012)? How does discordance between ex-ante perceptions and ex-post sensor measurements affect trust in sensors and scientists and policy makers? Commodore et al (2017) review a number of CBPR efforts and note that “air monitoring results, which reveal low levels of measured pollutants, may be less well received compared with results that are more confirmatory of suspicions.” They note that for CBPR, acceptance of results of air monitoring by all participants can be enhanced by making sure the research focused on an air quality issue of concern to the community, the communities are engaged throughout the design and implementation phases of the research, and that results are disseminated by the community (Commodore et al, 2017).
To date, the value of sensors has been appreciated mostly in terms of the data they produce as measurement devices. As the cost of sensors declines, individuals and communities may begin using them to do their own assessments of exposures (http://www.theinquirer.net/inquirer/feature/2432670/iot-could-lead-the-fight-against-poor-air-quality-in-london), in some cases with the goal of confirming perceived exposures, in other cases using the data to modify behaviors or activities to reduce exposures, or using the data to inform health studies. Whether these behavioral and social changes actually occur as sensors begin to be more widely available is ripe for scientific inquiry. Studies such as the Columbia University “Biking and Breathing” study are being designed and implemented to evaluate personal exposures measured using sensors (http://www.wnyc.org/story/biking-pollution-new-york-city-breathing/), but evaluations of how behaviors are modified by sensors, or how community decisions are affected by low cost sensors are not yet widespread. Sensors vary in quality, performance and ease of use (Jiao et al, 2016), and individuals or community organizations will benefit from easy to understand evaluations of different sensors that can guide the selection and use process. Several recent studies have evaluated low cost sensors and found that while promising, may available sensors suffer from inaccuracy and require frequent calibration, although for some pollutants, the readings are still useful for coarser statements about air quality, e.g. high vs low pollution levels, and can help communities understand local air quality (Castell et al, 2017; Rai et al, 2017). Understanding how individuals view their role as citizen scientists in both understanding local air quality and related health consequences will become increasingly important as communities seek to engage with policymakers to address their health concerns. In communities with historically higher exposures to contaminants, the level of trust in policymakers and the information on air quality provided by experts or the government may be lower, and those communities may seek to use air quality sensors to “interrogate” traditional sources of scientific information on air quality (Whatmore, 2009). Understanding whether communities fully understand limitations of sensors (e.g. accuracy) and their potential applications (e.g. not useful for regulatory compliance) can help direct resources toward better engagements with communities.
1.2.2.3. Behavioral, socio-political, and ethical implications of introducing sensors into communities
The introduction of sensors and sensor data can have a wide range of impacts on individuals and communities. Given the substantial variance in sensors currently available on the market – varying in pollutant types measured, how data are presented to the user, whether personal actions are suggested – there is substantial uncertainty in how individuals or communities and their resulting exposure may be modified by sensor technology. For example, people may change their commuting patterns or bike and pedestrian routes to avoid areas with apparent higher air pollution exposures (Hankey et al, 2017). But there are many other potential impacts across health, well-being, and social dimensions. These impacts may be either positive, through reduced individual exposures or better understanding of times and locations with high pollution or negative, if they result in less physical activity, higher levels of anxiety, or worsened disparities/inequality. In the event of incomplete or inaccurate information on air quality from sensors, misinformation may drive behaviors that result in greater rather than lesser exposure to air pollution. Though there is a large public health literature on behavior change and incentivizing healthy behaviors (e.g. Glanz et al, 1997), relatively few studies have examined how individuals and communities translate environmental data into behavior change or otherwise make decisions in accordance with environmental data (Oltra et al, 2017). A general question is how does access to personalized exposure information impact thoughts, attitudes and decisions? More specifically, would provision of expert advice on possible choices to reduce exposure provide confidence in making better decisions?
One important social dimension related to sensors is how they might change the dynamics between polluted communities, polluters and regulators (Powell, 2014). Availability of air quality sensors can change perceptions of responsibility for air pollution-related exposures (Monahan and Mokos, 2010), and research would be useful to understand these changes in perceptions in both affected communities and in those authorities traditionally responsible for reducing air pollution burdens. Sensor campaigns that convey either explicitly or implicitly that individuals are now responsible for any health effects that occur can paradoxically reduce the sense of empowerment for individuals who are not able to engage in protective behaviors or move away from adverse conditions. Conversely, collaborative approaches can increase public trust by engaging the public in the collection of environmental data and generation of environmental knowledge (Whatmore, 2009).
Beyond concepts of inclusion from the responsible innovation and risk governance frameworks, it is important to recognize how the representation of traditionally marginalized groups in the development and use of sensors can be hindered by interlocking systems of oppression, based on structural inequities such as race, class and gender (Dressel, et. Al 1997). This systemic oppression can often create a wall between these communities and developers, researchers, and regulators (Dressel, et. Al 1997). Considering this observation, it is important that the inclusion of these groups goes beyond tokenism and recognizes that there is significant heterogeneity within vulnerable populations (Dressel, et. Al 1997).
In a recent collaborative effort, the US EPA worked with the Ironbound Community Corporation (ICC) in Newark, New Jerseyto test an “Air Sensor Toolbox for Citizen Science.” (Kaufman et al, 2017) The ICC is a recognized environmental justice community as indicated by the EPA’s EJSCREEN tool, which shows a population with relatively high minority proportion (77 percent nonwhite), low education (93 percent with less than high school education) and income (76 percent low income) and relatively high environmental indicators (greater than 80th percentile for several indicators) (US EPA, 2017). The project successfully built a collaborative environment where community members worked alongside EPA scientists to characterize air pollution exposures in the community. Through this effort, the team together generated a mutual understanding of the strengths and weaknesses of the air pollution data collected with the lower-cost sensors, the potential uses of the data, and cases where the data would not be useful (e.g. in determining compliance with National Ambient Air Quality Standards). Calls for responsible innovation suggest the importance of including community voices in the development of a data collection protocol such as the one implemented in Newark. A key finding in this study was that the community members consistently sought to derive personalized meaning from the data through the lower-cost sensors. This is consistent with the theory of inclusive risk governance in that individual actors respond based on their own understanding of risk, and therefore will seek to understand risk by how air quality data can impact their own circumstances. One of the key principles of risk governance is communication and inclusion (van Asselt and Ren, 2011), and the challenges of data interpretation highlights the need for two-way communication between the air quality monitoring experts and community members. The US EPA is working to address some of these challenges by developing methods to translate sensor readings for ozone and PM2.5 into scales that relate to the AQI and provide guidance on behavioral responses when sensor readings are elevated (Mannshardt et al, 2017).
Those entities producing new knowledge, through data acquisition and analysis for example, are often those that are perceived as voices of authority. Sharing in the collection of air quality observations and co-generation of air quality knowledge can change the dynamics of interactions between citizens and institutions, and lead to co-management of risk (Berkes, 2009). The relationships between traditional knowledge generators (experts) and citizen (local) knowledge generators will need to evolve, and will require understanding the different perspectives of each group. While experts may be most concerned about the quality of the data used to generate knowledge, it will be informative to also study how non-expert individuals and groups view data quality and uncertainty regarding sensor data. How important to individuals and communities is the participatory nature of sensor data collection versus the higher risk of inaccurate data?
Lastly, the introduction of sensors may have profound ethical implications. Powell (2014) notes that the ability of new data generating technologies to improve the lives of citizens depends on several factors. These include the relationships between individuals and institutions, economic factors shaping technology development and distribution, and shared (or not) values for openness and transparency. It is still uncertain as to whether availability of low-cost sensors either alleviate or exacerbate disparities in health or economic status among population subgroups of varied incomes, ethnicity, or race. Low-cost sensors provide opportunities for marginalized communities to engage with new technologies and build knowledge about local air quality conditions and potentially the local sources that are contributing to poor air quality. This feeds new knowledge into discussions with state and local air quality planners to address local air quality concerns and improve community health for low-income or EJ communities. However, availability of sensors does not guarantee these outcomes. The basic nature of this problem is that while biomedical and environmental knowledge and technology creates the capacity for individuals and communities to improve their personal health, it can also exacerbate disparities in health outcomes between different populations. This paradox holds true for health outcomes attributable to air pollution exposures as well. For example, low income vs high income populations have different access to knowledge and technologies. If sensor technologies are still too costly for low income communities to use, or if the level of education and expertise needed to use the sensors is too great, sensors can benefit higher income, higher education populations to improve their exposures, while lower income, lower education populations do not receive the same benefit, leading to increased exposure and health disparities. Certain social factors, such as the ability to afford these technologies, time and willingness to engage, or technical and scientific proficiency, play a role in determining who has access to low-cost air quality sensors and the information they help generate (Gabrys, 2014). If the resulting information leads to a greater awareness of exposure to poor air quality conditions and opportunities to reduce those exposures, then those who are excluded from this process lose out on important health-related benefits. The idea that observed patterns of disease and death are affected by the social context of a population has been termed “social shaping” (Link, 2008), and research into this phenomenon presents an opportunity to introduce social science insights and methods into air quality and health science. Marriage of these sciences can bring about a better understanding of how the risk of disease and death reflects the interface of environmental and biological stressors with awareness and access to information and technologies, and can help identify the most effective types of interventions to alleviate health disparities. For example, programs that provide access to low-cost sensors, coupled with opportunities to engage with individuals or groups with a range of professional or academic backgrounds related to sensor technologies, environmental public health, and air quality policy, could increase equality in knowledge of local air quality in low-income areas, and result in localized programs to reduce exposures or policies to reduce emissions from sources contributing to locally elevated ambient air pollution. As another example, recognition of language barriers in communities with high proportions of non-English speaking populations could inform development of sensors with instructions, displays, and outputs in the community language. This can enable these communities to more actively understand local air quality conditions and engage with air quality planners to address sources and exposures of concern for their communities.
1.2.2.4. Assessing methods for communicating sensor data
There are a number of potentially important research approaches that can lead to more effective communication and improved understanding of how people respond to the availability of individualized, continuous air quality information. Several of these approaches involve consideration of how evidence is framed in risk messages (Gallagher and Updegraff, 2011; Rothman et al, 2006)
What are the most effective ways to communicate measurements obtained with air quality sensors? Measurements delivered as visual representations or as health messages may be more or less effective in altering individual behavior such that subsequent potentially harmful exposures are reduced. Whether messages are framed as improving health (gains) or preventing health damages (losses) can affect behavioral responses (Gallagher and Updegraff, 2011). The direction of the impact may be dependent on the nature of the behavioral response, with gain-framed messages likely to positively impact behaviors to prevent an exposure, and loss-framed messages likely to positively impact behaviors that identify when an exposure has occurred (Rothman et al, 2006). Visual approaches may differ in their effectiveness based on the desired application, e.g., should sensor results be communicated in different ways depending on whether they are used to achieve health risk reductions or for identification of emissions sources? Evaluations of the effectiveness of communication strategies should evaluate what is actually communicated to recipients (the message as they receive it), rather than just what is nominally displayed on a screen.
At the same time, research is needed beyond initial “messaging” to ascertain how people can be encouraged to make checking air quality a habit so they can realize long term benefits. Research on the phenomena of “attention fatigue” or “information fatigue” that can identify ways to reduce this fatigue will hopefully reveal technologies and tools to increase meaningful persistence of use that result in positive behavioral responses. It would also be useful to understand how behavioral responses to sensor data differ over time, i.e., in the short-term, sensor data may drive changes in daily personal activities and transportation choices (Hankey et al, 2017), while over a longer-term, people may geographically relocate, seek political action, or make investments in perceived health protection such as purchasing home filtration units. Understanding the persistence and effectiveness of long-term behavior changes will be important in informing the benefits of introducing low-cost sensors into communities.
The effectiveness of communication methods for sensor information likely varies with the intended audience. Measurable demographic and socioeconomic characteristics may help to explain utilization of sensor-based data and associated risk information, and can be used to shape the messaging to different populations. Research is needed on the optimal translation – in some cases simplification - of messages and data, either by the sensor itself or by experts. Simplifying the message may make sensor data more comprehensible, but also less interesting and less useful. Users may have different levels of numeracy, and decisions about the level of numerical information provided can shape the ultimate use by the community. Users may prefer apps that provide behavioral choices, e.g., different movement pathways, rather than prescription of activities, e.g. staying indoors (Zappi et al, 2012). There is an extensive literature on communicating health risk information that may be informative to future research on sensor-based messages (see for example Lundgren and McMakin, 2013; McComas, 2007).
On the other hand, consistent with today’s information-overloaded society, additional data may not be appreciated by some individuals. Research into the impact of continuous provision of air quality information on individual and community well-being or “happiness” can help guide decisions on how and when to deliver air quality information, e.g., does continuous data increase worry or enhance peace of mind? The health communication literature has reported that “information fatigue” or “information overload” can be detrimental to decision-making and cause stress and anxiety (Eppler, 2015), or lead to ignoring of relevant risk information (Revere et al, 2015). Little is known about these relationships with regard to air quality but as sensors further penetrate into personal use, their importance will become more evident. Mixed methods may be appropriate to address these questions, including use of on-line and mobile app-based surveys, focus groups, and interviews.
The types of communication may also depend on how users want to employ sensor data. For example, some individuals may be primarily concerned with exceeding identified health benchmark levels, while others may simply want to reduce their own personal air pollution exposures. Research is needed to understand the extent to which individuals can use sensors to “customize” their own air quality goals. Sensitive individuals, e.g. asthmatics, heavy outdoor exercisers, children, have the potential to use sensor data to determine where and when they have higher exposure or more severe responses to air pollution, and may want to know how to minimize those exposure and outcomes. Evaluating how much confidence the sensor user has in indices and associated messaging like the AQI may guide his or her interpretation of sensor messages related to perceptions of local air quality conditions. EPA has been cognizant of the emergence of sensor data at high time resolutions and has currently responded in two ways – 1) communicating that the official AQI should only be used as intended, for specific data sets and at averaging times matching the National Ambient Air Quality Standards, and 2) providing a proposed alternative color scale and data messaging that would be appropriate for high time resolution sensor data (https://www.epa.gov/air-research/communicating-instantaneous-air-quality-data-pilot-project).
Additionally, better understanding of how the success of different communication approaches is affected by levels of social vulnerability and behavior within groups is also needed. Are individuals and communities less likely to employ air quality sensors when they perceive the information provided is less actionable, i.e., there is nothing I can do about it? A reading of poor air quality can cause alarm, but if there are no appropriate mitigation options for an individual or community to address exposures and risks, individuals may become disinterested or develop a sense of powerlessness that may increase anxiety and reduce well-being.
Environmental health literacy is also critical to effective use of sensor data. Consideration of approaches to ensure that environmental health literacy is maintained in communities as new residents enter the community, and over time as children become adults and raise their own families. Developing messages based on sensor data that are targeted to sensitive populations such as children and the elderly and those with pre-existing heart and lung disease can improve the likelihood of behavioral changes that reduce exposures for these populations.
1.2.2.5. Harnessing crowdsourcing capabilities
As communities begin to crowdsource the analysis of their sensor data, social scientists can help to understand how different communities organize to share/generate knowledge from sensor networks. Social media or other web-based platforms open the possibility for individuals and communities to share their own analyses of sensor data. Other outlets for dissemination of data and results could include local sporting events, community events, religious events, etc. These data may have been personally collected or perhaps have been publicly provided by others. These options raise new challenges and opportunities for knowledge generation. A key question that results is “how do different user groups analyze and utilize sensor data (e.g. interpret, communicate, and respond to) compared with how scientists use the data, taking into account timing, delays, interpretation, and dissemination?”
Social scientists can help to understand the role of social analysis of data, e.g. crowdsourcing, community contextual analysis, team model building, etc., to better understand fundamentals of sensor use and those factors that enable exposure and risk reductions. Investigation into the extent that these types of social analytical tools engage stakeholders, expand their use, and aggregate as well as leverage results over temporal and spatial scales will enhance the improvement of advanced analytical tools for communities. Efforts in this regard have begun. The “Common Sense Community” air quality data analysis platform is an example of a system specifically designed to allow novice citizen scientists to analyze, visualize, and annotate data collected from sensors, within a collaborative software environment (Willet, 2012). EPA has introduced the Real Time Geospatial Data Viewer (RETIGO) web-based tool for visualization and analysis of sensor data (www.epa.gov/retigo). Another example is HabitatMap (http://habitatmap.org/) that allows users to upload measurements from the AirBeam sensor (http://www.takingspace.org/aircasting/airbeam/) and pinpoint air emissions facilities in NYC and alert public officials to potential environmental concerns.
1.3. Conclusions
Emerging air sensor technologies may significantly shape how individuals and communities perceive and respond to information about their air quality. Given the complexity of the science of air pollution, layered on the uncertainty in sensor data quality and interpretation, the downstream impacts of this technology on individuals and communities are uncertain. The spectrum of social sciences provides an array of metrics for judging human responses – at many levels of organization – to technologies and interventions (Figure 2). The social sciences can contribute to a better or more refined understanding of individual and community perceptions, attitudes, behaviors, and what determines various levels of engagement around and trust in air quality sensors and sensor data. Application of social science methods and tools to evaluate how sensors are entering the social arena and the resulting impacts is an area with high potential to enhance the quality of individual and community life.
The sensor itself is only the beginning of the process of understanding, communicating, and responding to air pollution. Other steps include processing and translating the sensor data, collecting ancillary information that might explain unusually high or low readings (e.g., taking photos of where the reading occurred), data aggregation and sharing, visual and statistical analysis, and visual display and communication. Social or other virtual networks can be a potentially valuable tool for users to compare, combine, and interpret sensor data gathered across a community. They can also foster data verification through group inquiry and group communication to regulatory and health officials. In the end, data collected through sensors can influence – positively or negatively – public attitudes and ultimately help inform policy decisions about air quality management.
Engaging in monitoring efforts can also increase environmental health literacy, which can lead to greater efficacy of interventions to improve community public health, and can deepen the expertise that community members have in areas typically considered to be the realm of academically-credentialed or technically-trained experts. The Ironbound community successfully learned how to assemble, install, operate and troubleshoot the air quality sensor pods throughout the course of the EPA research study (Kaufman et al, 2017). These skills are transferrable to future studies and raises the community’s capacity for understanding their own air quality. Beyond the technical skills, the community also gained knowledge of spatial and temporal variability of pollution as well as potential pollution sources. With the information gained, they plan to collaborate with researchers on future studies to further explore pollution sources, levels and exposures in their community. The Imperial county community air monitoring network project also led to increased capacity in the community to operate and maintain air quality sensors. However, they point to the continuing need for partnerships with government agencies and researchers, recognizing the important roles that each plays in ensuring credible, reliable, and meaningful analyses, and due to the potentially high costs of maintaining large sensor networks and the low available resources in many vulnerable communities. (English et al, 2017).
The integration of social science and technology development, which builds from the experiences of those engaged in citizen science and those working in academic and government social and natural science communities, will augment the value of air quality sensor information and the likelihood of developing successful solutions to public health problems posed by air pollution.
Acknowledgment:
There are no acknowledgments or grant support for this work.
Footnotes
The authors declare they have no actual or potential competing financial interest.
References
- Aoki P, Woodruff A, Yellapragada B, Willett W. 2017. Environmental Protection and Agency: Motivations, Capacity, and Goals in Participatory Sensing. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 3138–3150. 10.1145/3025453.3025667. [DOI] [Google Scholar]
- Bales E, Nikzad N, Ziftci D, Quick N, Griswold W, Patrick K. 2014. Personal Pollution Monitoring: Mobile Real-Time Air Quality in Daily Life. Unpublished manuscript. Available: http://cseweb.ucsd.edu/~earrowsm/TR.pdf [accessed 7 Nov 2017)
- Berkes F 2009. Evolution of co-management: Role of knowledge generation, bridging organizations and social learning. Journal of Environmental Management. 90: 1692–1702. 10.1016/j.jenvman.2008.12.001 [DOI] [PubMed] [Google Scholar]
- Castell N, Dauge FR, Schneider P, Vogt M, Lerner U, Fishbain B, Broday D, Bartonova A. 2017. Can commercial low-cost sensor platforms contribute to air quality monitoring and exposure estimates?, Environment International, 99: 293–302, 10.1016/j.envint.2016.12.007. [DOI] [PubMed] [Google Scholar]
- Cohn JP. 2008. Citizen Science: Can Volunteers Do Real Research?, BioScience, 58(3): 192–197. 10.1641/B580303 [DOI] [Google Scholar]
- Commodore A, Wilson S, Muhammad O, Svendsen E, Pearce J. 2017. Community-based participatory research for the study of air pollution: a review of motivations, approaches, and outcomes. Environ Monit Assess. 189(8):378 10.1007/s10661-017-6063-7 [DOI] [PubMed] [Google Scholar]
- Corburn J 2005. Street Science: Community Knowledge and Environmental Health Justice. MIT Press, Cambridge. ISBN: 9780262532723 [Google Scholar]
- de Wit JBF, Das E, Vet R. 2008. What Works Best: Objective Statistics or a Personal Testimonial? An Assessment of the Persuasive Effects of Different Types of Message Evidence on Risk Perception. Health Psychology. 27: 110–115. 10.1037/0278-6133.27.1.110. [DOI] [PubMed] [Google Scholar]
- Dressel P, Minkler M, Yen I. 1997. Gender, race, class, and aging: advances and opportunities. International Journal of Health Services, 27(4), pp.579–600. 10.2190/7XAY-PYBN-AA5L-3DRC. [DOI] [PubMed] [Google Scholar]
- English PB, Olmedo L, Bejarano E, Lugo H, Murillo E, Seto E, Wong M, King G, Wilkie A, Meltzer D, Carvlin G, Jerrett M, and Northcross A. 2017. The Imperial County Community Air Monitoring Network: A Model for Community-based Environmental Monitoring for Public Health Action. Environmental Health Perspectives. 125(7). 10.1289/EHP1772 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Eppler MJ. 2015. Information Quality and Information Overload: The Promises and Perils of the Information Age In Cantoni L, Danowski JA, ed. Communication and Technology. Walter de Gruyter GmbH & Co KG. ISBN 978–3-11–027135-5 [Google Scholar]
- Finn S, O’Fallon L. 2017. The Emergence of Environmental Health Literacy—From Its Roots to Its Future Potential. Environmental Health Perspectives. 125(4). 10.1289/ehp.1409337. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gabrys J 2014. Programming Environments: Environmentality and Citizen Sensing in the Smart City. Environment and Planning D: Society and Space. 32:30–48. 10.1068/d16812. [DOI] [Google Scholar]
- Gallagher KM, Updegraff JA. 2011. Health Message Framing Effects on Attitudes, Intentions, and Behavior: A Meta-analytic Review. Annals of Behavioral Medicine. 43:101–116. 10.1007/s12160-011-9308-7. [DOI] [PubMed] [Google Scholar]
- Gentner D, Stevens AL. 2014. Mental Models. Psychology Press, New York. ISBN: 978–0898592429 [Google Scholar]
- Glanz K, Lewis FM, Rimer BK. 1997. Health Behavior and Health Education: Theory, Research, and Practice. 2nd Ed. Jossey-Bass, San Francisco CA. ISBN: 978–0787996147 [Google Scholar]
- Hankey S, Lindsey G, Marshall JD. 2017. Population-level exposure to particulate air pollution during active travel: planning for low-exposure, health-promoting cities. Environmental Health Perspectives 125(4):527–534. 10.1289/EHP442. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Jiao W, Hagler G, Williams R, Sharpe R, Brown R, Garver D, Judge R, Caudill M, Rickard J, Davis M, Weinstock L, Zimmer-Dauphinee S, and Buckley K. 2016. Community Air Sensor Network (CAIRSENSE) project: evaluation of low-cost sensor performance in a suburban environment in the southeastern United States, Atmospheric Measurement Technology, 9: 5281–5292. 10.5194/amt-9-5281-2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kaufman A, Williams R, Barzyk T, Greenberg M, O’Shea M, Sheridan P 3, Hoang a, Ash C, Teitz A, Mustafa M, Garvey S. 2017. A Citizen Science and Government Collaboration: Developing Tools to Facilitate Community Air Monitoring. Environmental Justice, 10(2):1–11. 10.1089/env.2016.0044. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lewis AC, Lee JD, Edwards PM, Shaw MD, Evans MJ, Moller SJ, Smith KR, Buckley JW, Ellis M, Gillot SR, White A. 2016. Evaluating the performance of low cost chemical sensors for air pollution research. Faraday Discuss. 189:85–103. 10.1039/C5FD00201J. [DOI] [PubMed] [Google Scholar]
- Link B 2008. Epidemiological Sociology and the Social Shaping of Population Health. Journal of Health and Social Behavior, 49(4): 367–384. 10.1177/002214650804900401. [DOI] [PubMed] [Google Scholar]
- Lundgren RE, McMakin AH. 2013. Risk Communication: A Handbook for Communicating Environmental, Safety, and Health Risks. Fifth Edition. John Wiley and Sons, Hoboken, NJ. ISBN: 978–1-118–45693-4 [Google Scholar]
- Mannshardt E, Benedict K, Jenkins S, Keating M, Mintz D, Stone S, Wayland R. 2017. Analysis of short-term ozone and PM2.5 measurements: Characteristics and relationships for air sensor messaging. Journal of the Air & Waste Management Association. 67(4):462–474. 10.1080/10962247.2016.1251995. [DOI] [PubMed] [Google Scholar]
- McComas KA. 2006. Defining Moments in Risk Communication Research: 1996–2005. Journal of Health Communication, 11(1): 75–91. 10.1080/10810730500461091. [DOI] [PubMed] [Google Scholar]
- McElfish J, Pendergrass J, Fox T. 2016. Clearing the Path: Citizen Science and Public Decision Making in the United States. Woodrow Wilson Center Science and Technology Innovation Program. Research Series Volume 04 Available: https://www.wilsoncenter.org/sites/default/files/clearing_the_path_eli_report.pdf [accessed 11 May 2016] [Google Scholar]
- Monahan T, Mokos JT. 2010. Sensing Environmental Danger in the City. International Review of Information Ethics, 12: 20–26. [Google Scholar]
- Montibeller G, von Winterfeldt D. 2015. Cognitive and Motivational Biases in Decision and Risk Analysis. Risk Analysis, 35: 1230–1251. 10.1111/risa.12360. [DOI] [PubMed] [Google Scholar]
- Moore G 2014. Crossing the Chasm, Third Edition. HarperCollins Publishers, New York. ISBN: 9780062292988 [Google Scholar]
- Morgan MG, Fischhoff B, Bostrom A, Atman CJ. 2002. Risk communication: A mental models approach. Cambridge University Press, New York. ISBN: 978–0521002561 [Google Scholar]
- Oltra C, Sala R, Boso À, Asensio SL. 2017. Public engagement on urban air pollution: an exploratory study of two interventions. Environmental Monitoring and Assessment. 189(6):296 10.1007/s10661-017-6011-6. [DOI] [PubMed] [Google Scholar]
- O’Rourke D, Macey GP. 2003. Community Environmental Policing: Assessing New Strategies of Public Participation in Environmental Regulation. Journal of Policy Analysis and Management, 22(3): 383–414. 10.1002/pam.10138. [DOI] [Google Scholar]
- Owen R, Macnaghten P, Stilgoe J. 2012. Responsible Research and Innovation: From Science in Society to Science for Society, with Society. Science and Public Policy 39: 751–760. 10.1093/scipol/scs093. [DOI] [Google Scholar]
- Powell A 2014. “Datafication,” Transparency, and good governance of the data city, in O’Hara K and Nguyen C (eds.), Digital Enlightenment Forum Yearbook, London, IOS Press; 10.3233/978-1-61499-450-3-215. [DOI] [Google Scholar]
- Pritchard H, Gabrys J. 2016. From Citizen Sensing to Collective Monitoring: Working through the Perceptive and Affective Problematics of Environmental Pollution. GeoHumanities, 2: 354–371. 10.1080/2373566X.2016.1234355. [DOI] [Google Scholar]
- Rai AC, Kumar P, Pilla F, Skouloudis AN, Di Sabatino S, Ratti C, Yasar A, Rickerby D. 2017. End-user perspective of low-cost sensors for outdoor air pollution monitoring. Science of The Total Environment, 607–608: 691–705. 10.1016/j.scitotenv.2017.06.266. [DOI] [PubMed] [Google Scholar]
- Revere D, Calhoun R, Baseman J, Oberle M. 2015. Exploring bi-directional and SMS messaging for communications between Public Health Agencies and their stakeholders: a qualitative study. BMC Public Health. 15: 621 10.1186/s12889-015-1980-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Renn O, Schweizer PJ. 2009. Inclusive risk governance: concepts and application to environmental policy making. Environmental Policy and Governance, 19(3), 174–185. 10.1002/eet.507. [DOI] [Google Scholar]
- Rothman AJ, Bartels RD, Wlaschin J, Salovey P. 2006. The Strategic Use of Gain- and Loss-Framed Messages to Promote Healthy Behavior: How Theory Can Inform Practice. Journal of Communication. 56: S202–S220. 10.1111/j.1460-2466.2006.00290.x [DOI] [Google Scholar]
- Sadd JS, Morello-Frosch R, Pastor M, Matsuoka M, Prichard M, Carter V. 2014. The Truth, the Whole Truth, and Nothing but the Ground-Truth: Methods to Advance Environmental Justice and Researcher-Community Partnerships. Health Education & Behavior. 41(3): 281–90. 10.1177/1090198113511816. [DOI] [PubMed] [Google Scholar]
- Scott D, Barnett C. 2009. Something in the Air: Civic science and contentious environmental politics in post-apartheid South Africa. Geoforum, 40: 373–382. 10.1016/j.geoforum.2008.12.002. [DOI] [Google Scholar]
- Silvertown J 2009. A new dawn for citizen science. Trends in Ecology & Evolution, 24(9): 467–471, 10.1016/j.tree.2009.03.017. [DOI] [PubMed] [Google Scholar]
- Stilgoe J, Owen R, Macnaghten P. 2013. Developing a framework for responsible innovation. Research Policy 42(9):1568–1580. 10.1016/j.respol.2013.05.008. [DOI] [Google Scholar]
- U.S. Environmental Protection Agency (U.S. EPA). 2009. Integrated Science Assessment for Particulate Matter (Final Report). EPA-600-R-08–139F. National Center for Environmental Assessment – RTP Division, Research Triangle Park, NC: December Available at: <http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=216546>. Accessed October 20, 2017. [Google Scholar]
- U.S. Environmental Protection Agency (U.S. EPA). 2013b. Integrated Science Assessment of Ozone and Related Photochemical Oxidants (Final Report). EPA/600/R-10/076F. National Center for Environmental Assessment – RTP Division, Research Triangle Park: Available at: <http://cfpub.epa.gov/ncea/isa/recordisplay.cfm?deid=247492#Download>. Accessed October 20, 2017. [Google Scholar]
- U.S. Environmental Protection Agency (U.S. EPA). 2017. EJSCREEN: Environmental Justice Screening and Mapping Tool. Available at: https://www.epa.gov/ejscreen. Accessed October 20, 2017. [Google Scholar]
- van Asselt MBA, Renn O. 2011. Risk Governance. Journal of Risk Research, 14: 431–449. 10.1080/13669877.2011.553730. [DOI] [Google Scholar]
- Whatmore SJ. 2009. Mapping Knowledge Controversies: Science, Democracy and the Redistribution of Expertise. Progress in Human Geography 33: 587–598. 10.1177/0309132509339841. [DOI] [Google Scholar]
- Willett WJ. 2012. Tools & Strategies for Social Data Analysis. University of California at Berkeley; Technical Report No. UCB/EECS-2012–224. Available: http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-224.pdf [accessed 7 Nov 2017] [Google Scholar]
- Zappi P, Bales E, Park JH, Griswold W, Rosing TS. 2012. The CitiSense Air Quality Monitoring Mobile Sensor Node. IPSN’12, April 16–20, 2012, Beijing, China: Association for Computing Machinery (ACM) 978–1-4503–1227-1/12/04. Available: http://seelab.ucsd.edu/papers/Zappi_IPSN12.pdf [accessed 7 Nov 2017] [Google Scholar]