Skip to main content
NCBI home page
Heliyon logoLink to Heliyon
. 2019 Aug 5;5(8):e02128. doi: 10.1016/j.heliyon.2019.e02128

Multi actor multi criteria analysis (MAMCA) as a tool to build indicators and localize sustainable development goal 11 in Brazilian municipalities

Ana CL Almeida 1,∗
PMCID: PMC6695285  PMID: 31428710

Abstract

The Sustainable Development Goals (SDGs) were adopted in 2015 to be a guideline to promote sustainable development through partnerships. Goal 11 was designed to improve the quality of life in cities however, confronted local governments with new challenges to delivery services and increase citizen participation. In this study, a conceptual framework was developed, and distinct indicators were analyzed to facilitate the implementation of SDG 11 in Brazilian municipalities. Two case studies were deployed based on the Multi-Actor Multi-Criteria Analysis (MAMCA) to guarantee stakeholder participation during the whole process. The results brought to light important challenges to urbanization at local levels. Also, the results suggest shifts in the ongoing model to evaluate the implementation of SDG 11 in Brazil. Governance, transparency and social participation were identified as critical issues to be addressed. Moreover, it will be necessary to adjust existing indicators and organize a consistent and frequent method to evaluate progress towards achieving SDG 11 targets at local levels. Integration of plans and policies related to climate change and disaster risk reduction represents another challenge to be faced by local governments. Finally, MAMCA can be a helpful tool to support local decision-makers to implement SDG 11 based on a multi-stakeholder view.

Keywords: Environmental science, Environmental analysis, Environmental assessment, Environmental management, Sustainable development, Sustainable cities, Planning, Urban sustainability, Agenda 2030

1. Introduction

Localising is the process of considering the subnational context in the achievement of Agenda 2030 and comprises two main processes: planning and implementing the SDGs, and monitoring SDG progress (United Nations Development Programme, UN-Habitat, & Global Taskforce of Local and Regional Governments, 2016). From the 17 Sustainable Development Goals (SDGs), Goal 11, “Make cities and human settlements inclusive, safe, resilient and sustainable” is considered as a lynchpin for local governments (United Nations Development Programme et al., 2016). Goal 11 targets relevant urban issues, such as housing, transportation, water, and air quality, development planning and integration between national and regional plans, and international agreements. The United Nations (UN) adopted a global indicator framework comprising 232 indicators to support SDG's achievement (United Nations General Assembly, 2017). Those indicators should be aligned to current local or regional indicators to facilitate the analysis of progress of SDGs (United Nations, 2015).

Indeed, sustainable city indicators are useful tools for monitoring, for evaluating progress of sustainability in the cities, and for fostering active social participation (Hiremath et al., 2013; Pupphachai and Zuidema, 2017; Shen et al., 2011; Tran, 2016). However, there are many challenges in selecting an appropriate indicator framework for measuring progress towards sustainable development, especially at local levels (Yang et al., 2017).

Many scholars have developed indicator tools and indexes from the perspectives of economics, society, and the environment to assess sustainability in the cities (Alkhalidi et al., 2018; Braulio-Gonzalo et al., 2015; Costa et al., 2007; European Commission DG Environment, 2018; Florissi, 2009; Hiremath et al., 2013; Munier, 2011; Pupphachai and Zuidema, 2017; Shen et al., 2011; Xiang, 2017). Also, international organisations such as World Bank, European Commission, Organisation for Economic Co-operation and Development (OECD), and UN-Habitat are working on indicators to assess urban development sustainability. The advantages and disadvantages of 28 toolkits were determined when assessing sustainable cities (Science for Environment Policy, 2018). Among the limits of assessment, the definition of a set of indicators covering the totality of the urban picture to assess sustainable development (Gómez-Álvarez et al., 2018) and the lack of comparison between the distinct systems of indicators (Keirstead, 2007; Yang et al., 2017) seem to be the most challenging limitations. Regarding the SDGs, the lack of well-designed conceptual frameworks for selecting indicators can result in ambiguous and confusing assessments (Hák et al., 2016).

In this research, national and local indicators were analysed to facilitate the implementation of SDG 11 in Brazilian municipalities, based on the Multi-Actor Multi-Criteria Analysis (MAMCA). MAMCA was adopted to provide a broader stakeholder approach in the selection of indicators. Multi-Criteria Analysis (MCA) has been used as a decision analysis technique for several decades and as well as with methodologies for thousands of applications (Greco et al., 2016). MAMCA is a type of multi criteria analysis (MCA) technique that has gained much popularity over the past decades (Kourtit et al., 2014) and has been used to enhance policy analysis by explicitly considering the opinions of various stakeholders (Kourtit et al., 2014; Macharis et al., 2009; Stathopoulos et al., 2012; Sun et al., 2015). The methodology differs from the classical approach of MCA because it allows the introduction of stakeholders at a very early stage of methodology (Bergqvist et al., 2015), takes into consideration all stakeholders' opinions and priorities, and facilitates the identification of key indicators and their linkages (De Brucker, Macharis and Verbeke, 2013).

In Brazil, perhaps, one of the main factors that will be an essential part of localising SDG 11 is developing guidance for municipalities implementing the targets and mapping out their capacity to deliver sustainability (Almeida et al., 2018). Regarding indicators, the Brazilian Institute of Geography and Statistics (IBGE) is developing a set of indicators to monitor the SDGs, taking into consideration the official list of indicators adopted by the United Nations (Instituto Brasileiro de Geografia e Estatistica, 2018). However, this initiative is focused on the national level rather than local level.

The present research was carried out in Brazil to support the municipal government in the implementation of SDG 11. The first part of this research aims to analyse existing sustainable urban indicators and their contribution to the achievement of SDG 11 at local levels, regarding a more comprehensive and holistic view of urban sustainability. The second part focuses on multi-stakeholder analysis with the aim of integrating distinct stakeholder visions into the selection of indicators. The results reflect relevant urban issues and offer a set of key indicators that can contribute to the performance evaluation of Brazilian municipalities in terms of SDG 11 achievement.

2. Main text

2.1. Methodology

2.1.1. Study areas

This research comprises two distinct geographic areas: Niteroi city, state of Rio de Janeiro, and the Inter-Municipal Consortium of the Paranapanema Valley (CIVAP) that is a cooperation arrangement between 27 municipalities located in Sao Paulo state (Fig. 1).

Fig. 1.

Fig. 1

Open in a new tab

Location of Niteroi city and Inter-Municipal Consortium of the Paranapanema Valley (CIVAP).

The choice of those localities was purposeful. Niteroi is a big city that has faced many urban issues due to rapid urbanization and local government has already developed participatory planning processes as a tool to improve sustainability in the city notably, the strategic plan Niteroi que Queremos 2033 (Prefeitura de Niterói, 2017). In contrast, CIVAP is a cooperation arrangement between 26 small municipalities and one big city and none of the municipalities have experience in strategic planning with social participation. Niteroi city is in the metropolitan area of Rio de Janeiro State, has an estimated population of 511,786 inhabitants and covers an area of 134,074 km2 (Instituto Brasileiro de Geografia e Estatistica, 2015). The total population lives in the urban area, which represents a challenge for promoting local sustainable development. In CIVAP area most municipalities are small, and the region has an estimated population of 1.207,105 inhabitants, covering an area of 9.101,718 km2 (Instituto Brasileiro de Geografia e Estatistica, 2017). Table 1 shows differences and similarities between the municipalities evaluated in this research, according to available social and economic development indicators.

Table 1.

Social and economic characteristics of municipalities evaluated in this research.a,b

State** Municipality Population (2010) Population estimate (2018) GDP per capita* (2010) GDP per capita* (2016) HDI (2010) Income Gini Coefficient (2010)
RJ 1 Niterói 487.562 511.786 23011.46 46202.31 0.837 0.5983
SP 2 Assis 95.144 103.666 14271.72 27921.64 0.805 0.5040
SP 3 Bastos 20.445 20.954 17711.81 39033.19 0.751 0.4175
SP 4 Bora 805 836 40546.42 100029.91 0.746 0.4002
SP 5 Campos Novos Paulista 4.539 4.932 18298.26 28307.64 0.706 0.4379
SP 6 Cândido Mota 29.884 31.212 16692.47 29350.05 0.747 0.4240
SP 7 Cruzalia 2.274 2.100 29208.86 31800.1 0.774 0.4393
SP 8 Echapora 6.318 6.141 14468.87 20518.61 0.745 0.5424
SP 9 Florínia 2.829 2.699 43182.29 43838.84 0.713 0.4944
SP 10 Ibirarema 6.725 7.663 16851.58 47003.22 0.708 0.4352
SP 11 Iepê 7.628 8.124 18280.42 30175.51 0.736 0.4632
SP 12 Joao Ramalho 4.150 4.495 15081.30 35767.38 0.741 0.4235
SP 13 Lutécia 2.714 2.663 18819.34 32273.06 0.720 0.4316
SP 14 Maracaí 13.332 13.967 32163.83 36926.11 0.771 0.4255
SP 15 Nantes 2.707 3.103 34462.89 31673.75 0.714 0.3530
SP 16 Ocaucu 4.163 4.287 14074.49 22459.31 0.717 0.3956
SP 17 Oscar Bressane 2.537 2.602 12419.03 32076.14 0.749 0.4496
SP 18 Palmital 21.186 22.168 22037.14 27670.81 0.746 0.4648
SP 19 Paraguacu Paulista 42.278 45.455 21844.79 26291.19 0.762 0.4641
SP 20 Paulistania 1.779 1.832 16910.60 16802.89 0.718 0.4293
SP 21 Pedrinhas Paulista 2.940 3.085 28143.45 33024.83 0.774 0.5619
SP 22 Pirapozinho 24.694 27.295 15913.46 26952.95 0.776 0.5286
SP 23 Platina 3.192 3.521 19813.06 22182.07 0.719 0.4960
SP 24 Quatá 12.799 14.006 40761.77 32791.62 0.738 0.4175
SP 25 Rancharia 28.804 29.688 29399.97 38668.19 0.751 0.4694
SP 26 Santa Cruz do Rio Pardo 43.921 47.395 22175.53 42021.86 0.762 0.4639
SP 27 Taciba 5.714 6.240 52288.24 21170.25 0.723 0.3861
SP 28 Taruma 12.885 14.812 30087.95 78512.45 0.753 0.5488
Open in a new tab

Note. *GDP in Reais (R$). **In urban areas.

a

National Bureau of Statistics of Brazil. https://cidades.ibge.gov.br/.

b

Information Technology Department of the Public Health Care System. http://tabnet.datasus.gov.br/cgi/ibge/censo/cnv/ginibr.def.

2.1.2. Research method

Qualitative research was designed to support the municipal government in implementing SDG 11, combined with a bottom-up approach to developing indicators. A conceptual framework composed of two stages was devised and is summarized in Fig. 2. First, a review of existing national and local indicators in the field of sustainable urbanization was conducted. At the local level, four major systems were selected: (1) the set of indicators adopted by the strategic plan Niteroi que Queremos 2033 (Gomes and Reys, 2015), (2) the set of indicators adopted by the Municipal Management's Effectivity Index (Tribunal de Contas do Estado de Sao Paulo, 2014), (3) SDG Mandala (Confederação Nacional de Municípios, 2017), and (4) the Sustainable Cities Program Indicators (Programa Cidades Sustentaveis, 2017).

Fig. 2.

Fig. 2

Open in a new tab

Conceptual framework for evaluating the implementation of SDG 11 in Niteroi City.

The strategic plan comprises a total of 57 strategies and 29 timely data indicators for monitoring the performance of local priorities in Niteroi city. The Municipal Management's Effectivity Index was launched in 2015 and comprises a total of 152 indicators. These indicators are classified into seven dimensions: education, health, planning, fiscal management, environment, protection of the cities, and information technology governance. The Index aims to promote the effectiveness of municipal management with transparency and accountability. SDG Mandala is a tool comprising 26 indicators and developed to support local public managers to monitor and evaluate the performance of Brazilian municipalities regarding the achievement of SDGS and Agenda 2030. The indicators are classified into 4 categories of sustainable development: institutional, economic, social, and environmental. The Sustainable Cities Program Indicators aims to improve municipal administration and is composed of 260 indicators categorized by SDG goals. From this total, 29 indicators were developed to assess the implementation of SDG 11 at local levels.

At national level, other two systems were identified: The Sustainable Development Indicators and SDGs Indicator. The Sustainable Development Indicators were developed by IBGE in 2002 for monitoring sustainable development and is composed of 132 indicators, also classified in the 4 categories of sustainable development. The SDGs Indicator is an information system, still under construction, for monitoring the implementation of Agenda 2030 in Brazil and is composed of a total of 244 indicators, 15 of them developed to monitor SDG 11 deployment.

Considering that indicators can be described or measured in different ways, the selection of indicators (first set) was based on six criteria: (1) easy to apply; (2) easy to calculate; (3) relevance to local level; (4) existence of available data; (5) contribution to the achievement of SDG 11 targets; (6) time-bound. The contribution to SDG 11 was the criteria adopted to exclude indicators concerning the strategic plan, SDG mandala, and sustainable development indicators. The other two systems were already organized through SDG 11 targets; thereby, all indicators were analyzed.

In the second stage, two exploratory case studies (Yin, 2003) were designed to contribute to the implementation of SDG 11 in Niteroi city and CIVAP region. The MAMCA consisting of seven key steps was adopted in this research to develop the case studies (Macharis et al., 2004). Steps 1 to 4 were developed during participatory decision-making workshops aimed at discussing the main challenges and opportunities regarding the forthcoming implementation of SDG 11 at local levels. These workshops were carried out in Niteroi city and CIVAP region in June 2017. In Niteroi city the workshop comprised the participation of 16 key stakeholders aggregated into 4 groups: eight from municipal government, three from business sector, two from universities, and three from state audit office. In the CIVAP region, the workshop involved the participation of 15 key stakeholders aggregated into 2 groups: ten from municipal government and five from scientific and technological community. It is important to highlight that two representatives from the business sector participated in the workshop carried out in the CIVAP region. However, they did not want to take part in the evaluation. Informed consent was obtained from all workshop participants. Ethical approval was obtained from the Griffith University Human Research Ethics Committee in accordance with the National Statement on Ethical Conduct in Human Research (GU 2016/748).

As shown in Fig. 3, the first step of MAMCA methodology is to define the problem and alternatives for evaluation. In this research, SDG 11 targets are chosen as preselected alternatives and are submitted for stakeholder evaluation during the workshop. The second step in the MAMCA involves the definition of the range of stakeholders to be consulted. Selecting the range of key stakeholders was an early challenge. Many techniques have been developed to define a representative. The snowball sampling was the method adopted in this research to identify relevant stakeholders (Berg, 2001). This strategy was the best way to locate key stakeholders in different Municipalities. In the step 3, stakeholder groups identify objectives and criteria. The criteria were not tracked by the literature (Turcksin et al., 2011), but identified during the workshops, based on interactive discussions between the stakeholders. Stakeholder groups identified a specific set of criteria and allocated weights to each distinct criterion. The weights are determined by the importance the stakeholders give to each criterion and distinct weighting methods can be adopted as direct weights, direct allocation, and so on (Grafakos et al., 2015). This approach solves the problem related to the (in)dependency of the criteria (Macharis et al., 2009). A weight scale ranging from 1 to 9, with 1 meaning “not so important” and 9 “extremely important”, was adopted to help stakeholders to identify their preferences. It is important to note that the same value could be used to distinguish criteria if they had the same importance. Also, considering that a stakeholder group was constituted by distinct members, the geometric mean was calculated (Turcksin et al., 2011). Step 4 consists of constructing indicators to evaluate each individual criterion and defining the measurement method. Steps 1 to 4 are considered mainly analytical and anticipate the overall analysis (Macharis et al., 2012).

Fig. 3.

Fig. 3

Open in a new tab

Schematic diagram of the seven key steps of Multi Actor Multi Criteria Analysis (MAMCA). Source: adapted from Macharis et al. (2004).

For the fifth step, the Analytical Hierarchy Process (AHP) was used to obtain the relative importance of the criteria identified by the four stakeholder groups simultaneously (Saaty, 1980). Within AHP, the different measures or alternatives are compared to support the decision-maker in the final decision. A multi-criteria decision analysis (MCDA) is performed for each stakeholder group. In step 6, criteria are analyzed to provide a multi-stakeholder view on the different alternatives. Step 7 is referred as the implementation phase. This is the final step of MAMCA and compile the points of view of each stakeholder. Based on the information gathered from the previous steps, the local government would decide the alternatives to be implemented (Macharis et al., 2010).

2.2. Results

Existing national and local indicators were evaluated according to six criteria as shown in Tables 1 and 2, respectively. Concerning national indicators, all data are collected by IBGE, and a total of 24 quantitative indicators were selected (Table 2). All the selected indicators are easy to apply, easy to calculate, relevant to local levels, and contribute to achievement of SDG 11 targets. Nevertheless, data availability and time-bound factors can be constraints to the analysis of progress of SDG 11 implementation. Most of the data required for SDGs indicators is not readily available because the system is under construction. Concerning the sustainable development indicators, the main barrier is that the indicators are not frequently updated in order to facilitate monitoring and evaluation.

Table 2.

Selection of existing national indicators.

System Theme Indicators Criteria
c.1 c.2 c.3 c.4 c.5 c.6
Sustainable Development Indicators Air quality Annual mean levels of fine particulate matter (i.e. PM10) ✓ ✓ ✓ 2011 ✓ ♦
Housing Percentage of adequate housing ✓ ✓ ✓ 2015 ✓ ♦
Participatory planning and management Number of civil society organizations per 100,00 inhabitants ✓ ✓ ✓ 2010 ** ♦
Number of municipalities with environmental council ✓ ✓ ✓ 2013 ** ♦
Percentage of public consortium ✓ ✓ ✓ 2015 ** ♦
Safety Number of deaths per 100,000 inhabitants ✓ ✓ ✓ 2012 ✓ ♦
Transportation Number of deaths due to traffic accidents ✓ ✓ ✓ 2012 ** ♦
Waste management Percentage of solid waste regularly collected ✓ ✓ ✓ 2015 ✓ ●●
Percentage of solid waste with adequate final discharge ✓ ✓ ✓ 2008 ✓ ♦
Number of municipalities with adequate waste final discharge ✓ ✓ ✓ 2008 ** ♦
SDGs Indicators Air quality Annual mean levels of fine particulate matter (e.g. PM2.5 and PM10) in cities (population weighted) ✓ ✓ ✓ ○ ✓ ♦
Cultural and natural heritage Total expenditure (public and private) per capita spent on the preservation, protection and conservation of all cultural and natural heritage, by type of heritage (cultural, natural, mixed and World Heritage Centre designation), level of government (national, regional and local/municipal), type of expenditure (operating expenditure/investment) and type of private funding (donations in kind, private non-profit sector and sponsorship) ✓ ✓ ✓ ○ ✓ ♦
Disaster risk reduction Number of deaths, missing persons and persons affected by disaster per 100,000 people ✓ ✓ ✓ 2015 ✓ ●●
Direct disaster economic loss in relation to global GDP, including disaster damage to critical infrastructure and disruption of basic services ✓ ✓ ✓ ○ ✓ ♦
Housing Percentage of urban population living in slums, informal settlements or inadequate housing ✓ ✓ ✓ 2010 ✓ ●●
Integrated policies and plans related to climate change and disaster risk reduction Proportion of local governments that adopt and implement local disaster risk reduction strategies in line with the Sendai Framework for Disaster Risk Reduction 2015–2030 ✓ ✓ ✓ 2013 ✓ ♦
Number of countries with national and local disaster risk reduction strategies ✓ ✓ ✓ ○ ✓ ♦
Participatory planning and management Ratio of land consumption rate to population growth rate ✓ ✓ ✓ ○ ✓ ♦
Proportion of cities with a direct participation structure of civil society in urban planning and management that operate regularly and democratically ✓ ✓ ✓ ○ ✓ ♦
Integrated planning Proportion of population living in cities that implement urban and regional development plans integrating population projections and resource needs, by size of city ✓ ✓ ✓ ○ ✓ ♦
Sustainable and resilient buildings Proportion of financial support to the least developed countries that is allocated to the construction and retrofitting of sustainable, resilient and resource-efficient buildings utilizing local materials ✓ ✓ ✓ ○ ✓ ♦
Transportation Percentage of population with adequate access to public transport, by sex, age and persons with disabilities ✓ ✓ ✓ ○ ✓ ♦
Waste management Proportion of urban solid waste regularly collected and with adequate final discharge out of total urban solid waste generated, by cities ✓ ✓ ✓ ○ ✓ ♦
Open in a new tab

c.1 - Easy to apply; c.2 - Easy to calculate; c.3 - Relevant to local levels; c.4 - Latest available data; c.5 - Contributes to SDG 11 target; c.6 - Time-bound. (a)- Data available only for a few metropolitan regions.

✓ - Indicator fits criteria; ** - Indicator is not explicit in the SDG official list, but contributes to SDG 11 targets; ○ - No data available; ●● - Annual; ♦ - Frequency unknown.

Regarding the local systems (Table 3), a total of 53 indicators was selected, both quantitative and qualitative. All the indicators are easy to apply, easy to calculate, and relevant to local levels. The Municipal Management's Effectivity Index is skewed towards preventing disasters and waste management improvements. The system can be applied to all municipalities and should be updated yearly. The SDG Mandala is the system with fewer indicators for evaluating the implementation of SDG 11 and was developed to be updated yearly, which can be a problem because some crucial data for monitoring evolutions are not frequently collected. Concerning the Sustainable Cities Program, the system covers almost the totality of SDG 11 targets and is skewed towards improving transportation. However, as mentioned, some flaws with the measurements can be noted because some data are not readily available and/or not frequently updated. Also, some indicators have not been previously tested or implemented in any municipality.

Table 3.

Analysis of existing local indicators.

System Theme Indicators Criteria
c.1 c.2 c.3 c.4 c.5 c.6
Municipal Management's Effectivity Index Disaster risk reduction Disaster risk Assessment ✓ ✓ ✓ 2016 ** ●●
Integrated Information System for Disaster Management ✓ ✓ ✓ 2016 ** ●●
Municipal coordination of emergency operations ✓ ✓ ✓ 2016 ** ●●
Municipal infrastructure to organize occurrences ✓ ✓ ✓ 2016 ** ●●
Resilient cities ✓ ✓ ✓ 2016 ** ●●
Training and contingency planning ✓ ✓ ✓ 2016 ** ●●
Urban mobility Urban mobility plan ✓ ✓ ✓ 2016 ** ●●
Waste management Adequate final discharge of total solid waste generated ✓ ✓ ✓ 2016 ✓ ●●
Adequate discharge of civil construction waste ✓ ✓ ✓ 2016 ** ●●
Integrated solid waste management plan in compliance to the National Solid Waste Policy ✓ ✓ ✓ 2016 ** ●●
Quality of landfill management ✓ ✓ ✓ 2016 ** ●●
SDG Mandala Green areas Number of municipal conservation units ✓ ✓ ✓ 2015 ** ♦
Safety Number of deaths per 100,000 people ✓ ✓ ✓ 2018 ✓ ●●
Waste management Percentage of the population with adequate waste collection service ✓ ✓ ✓ 2015 ✓ ♦
Sustainable Cities Program Indicators Cultural and natural heritage Local cultural heritage council ✓ ✓ ✓ 2006 ** ♦
Municipal legislation to deal with environmental issues ✓ ✓ ✓ ○ ** ♦
Share of municipal budget with cultural heritage ✓ ✓ ✓ ○ ✓ ●●
Actions on the preservation, protection and conservation of all cultural and natural heritage ✓ ✓ ✓ ○ ** ♦
Disaster risk reduction Percentage of households in risk areas ✓ ✓ ✓ 2010(a) ** From 2020
Local actions or risk management instruments ✓ ✓ ✓ 2017 ** ♦
Economic and physical losses due to natural hazard events ✓ ✓ ✓ 2018 ✓ ●●
Number of injuries and deaths due to disasters ✓ ✓ ✓ 2018 ✓ ●●
Housing Housing deficit ✓ ✓ ✓ 2018 ** ●●
Percentage of urban population living in inadequate housing ✓ ✓ ✓ 2018 ✓ ●●
Percentage of rented households ✓ ✓ ✓ 2018 ** ●●
Participatory planning and management Municipal public consortium, partnerships, business sector and/or community support ✓ ✓ ✓ ○ ** ♦
Integrated planning between all municipal Secretaries ✓ ✓ ✓ ○ ** ♦
Safe, green and public spaces Total of green area per inhabitant ✓ ✓ ✓ 2010 ✓ ♦
Percentage of inhabitants living within 300 metres of a green public area ✓ ✓ ✓ ○ ** ♦
Development, implementation and/or review of Master Plan ✓ ✓ ✓ 2017 ** ♦
Sustainable and resilient buildings Percentage of buildings constructed and retrofitted with criteria of sustainability ✓ ✓ ✓ ○ ✓ ♦
Urban mobility Extension of cycle lanes (km) ✓ ✓ ✓ ○ ** ♦
Congestion and delays monitoring system ✓ ✓ ✓ 2016(b) ** ♦
Percentage of bus lanes that are meant exclusively for buses ✓ ✓ ✓ ○ ** ♦
Economic losses due to traffic accidents ✓ ✓ ✓ 2016 ** ♦
Accessibility for mobility impaired groups ✓ ✓ ✓ ○ ✓ ♦
Share of municipal budget with public transportation ✓ ✓ ✓ ○ ** ●●
Percentage of inhabitants living within 300 metres from public transportation ✓ ✓ ✓ ○ ✓ ♦
Affordability of public transport ✓ ✓ ✓ ○ ** ♦
Average daily travel time ✓ ✓ ✓ 2010 ** ♦
Percentage of public transport using renewable energy ✓ ✓ ✓ ○ ** ♦
Percentage of accessible sidewalks (Km) ✓ ✓ ✓ ○ ** ♦
Waste management Integrated solid waste management plan in compliance to the National Solid Waste Policy ✓ ✓ ✓ 2013 ** ♦
Strategic Plan Cultural heritage Number of licenses to construct and/or restore the city centre ✓ ✓ ✓ 2013 ** 2016/2020/2033
Disaster risk reduction Number of inhabitants living in risk areas ✓ ✓ ✓ 2014 ✓
Green areas Restoration of degraded green areas ✓ ✓ ✓ ○ **
Housing Proportion of urban population living in inadequate housing ✓ ✓ ✓ 2010 ✓
Housing deficit (%) ✓ ✓ ✓ 2010 **
Safety Number of deaths per 100,000 people ✓ ✓ ✓ 2012 ✓
Number of robberies and larceny victims per 100,000 people ✓ ✓ ✓ 2013 **
Sustainable urbanization Population density at city centre ✓ ✓ ✓ 2010 **
Transportation Average commute time ✓ ✓ ✓ 2010 **
Extension of bicycle lanes ✓ ✓ ✓ ○ **
Open in a new tab

c.1 - Easy to apply; c.2 - Easy to calculate; c.3 - Relevant to local levels; c.4 - Latest available data; c.5 - Contributes to SDG 11 target; c.6 - Time-bound. (a)- Data available only for some municipalities; (b) - Data available for only 15 cities.

✓ – Indicator fits criteria; ** - Indicator is not explicit in the SDG official list but contributes to SDG 11 targets; ○ - No data available; ●● - Annual; ♦ - Frequency unknown.

Finally, regarding the indicators of the strategic plan, it already included some SDG 11-related indicators. Importantly, although many indicators are based on data not readily available, all required data can be handled by municipalities. No local system includes indicators to evaluate integrated planning strategies to support positive economic, social, and environmental development. The integration of plans and policies related to climate change and disaster risk reduction is also not considered by any system.

Regarding the stakeholders' view on the important urban issues, a total of 14 criteria were identified by the four stakeholder groups in Niteroi City and eight criteria were identified by the two stakeholder groups in CIVAP region. Those criteria were used to build the value tree (Fig. 4). Not surprisingly, the analysis of the value tree reveals that there are important issues regarding the implementation of SDG 11. Some of those issues are directly related to priority areas identified in the the strategic plan Niteroi que Queremos 2033. However, except for the need to increase social participation and to delivery efficient public policies, the results show that each stakeholder group has a distinct opinion about the most important issues. The lack of convergence between the key stakeholders also makes it difficult to align priorities with ongoing efforts promoted by the municipal government.

Fig. 4.

Fig. 4

Open in a new tab

Value tree for each stakeholder category regarding the implementation of SDG 11 in Niteroi City and CIVAP region.

Concerning the indicators, a total of 31 indicators was identified by key stakeholders and are outlined in Table 4. Those indicators are comprehensive and incorporate a multi-stakeholder view about the main challenges to implementing SDG 11. It is also important to note that, although it was not the intention of this research to analyze trade-offs or interlinkages between SDGs, seven indicators identified by stakeholders are related to other goals, mainly Goal 17 (Partnerships and means of implementation) and Goal 16 (Peace, justice, and strong institutions). Furthermore, the results indicate that some qualitative indicators should be added for more accurate evaluation of SDG 11.

Table 4.

Set of indicators proposed by key stakeholders regarding the implementation of SDG 11 in Niteroi City and CIVAP region.

Stakeholder Group Criteria Indicators SDG Target
Municipal Government 1. Disaster risk reduction 1.1 Adequate methodology to improve infrastructure in slums 11.1
1.2 Map risks to slums upgrading 11.5
1.3 Slums upgrading based on participatory and integrated approach 11.3
1.4 Proportion of financial support allocated to upgrade slums 11.1
2. Housing 2.1 Planned settlements 11.1
2.2 Citizen participation and social control 11.3
2.3 Housing finance 11.1
3. Institutional 3.1 Efficient public services SDG 16
4. Legislation 4.1 Effective application ​of the law and regulations SDG 16
5. Social participation 5.1 Enhance community participation SDG 17
6. Transportation 3.1 Citizens' satisfaction with public transport 11.2
3.2 Modal Split 11.2
3.3 Review of bus concession contracts 11.2
3.4 Proportion of population with convenient access to public transport∗ 11.2
7. Waste management 4.1 Efficient waste management services 11.6
4.2 Regional public landfills 11.6
Universities/Scientific and technological community 1. Efficiency of public policies 1.1 Citizens' satisfaction with public services SDG 17
2. Finance and investments 2.1 Share of municipal budget with education, research, and development SDG 17
3. Partnerships 3.1 Establish partnerships with development agencies SDG 17
3.2 Establish partnerships with business sector SDG 17
4. Planning 4.1 Citizens' satisfaction with opportunities to participate in local planning and decision-making process 11.3
5. Quality of Education 5.1 Improve the quality of public schools SDG 4
6. Social participation 6.1 Projects to enhance social participation SDG 17
6.2 Promote community engagement SDG 17
Business Sector 1. Access to basic services 1.1. Access to improved water 11.1
2. Economic and social development 2.1. Promote new alternatives to integrate economic and social development SDG 8
3. Efficiency of public services 3.1 Citizens' satisfaction with public services SDG 17
4. Social Participation 4.1 Dialogue between stakeholders 11.3
State Audit Office 1. Communication 1.1 Mobilize civil society to communicate and raise awareness about SDGs goals SDG 16
2. Integration 2.1 Integration of systems to enhance social control SDG 16
3. Transparency and social control 3.1 Existing mechanisms and processes which ensure openness and accountability SDG 16
Open in a new tab
∗

Indicator should be disaggregated by gender, age, and persons with disabilities.

Thereby, it can be said that while municipal government is focused on delivery services in three critical areas, the general population is more interested in monitoring the quality of services, transparency and social participation.

Based on the AHP results, Fig. 5 shows the preference of local stakeholder groups. In Niteroi city, reducing disaster risks and improving transportation were the most preferred criterion by municipal government representatives. Moreover, local public authorities also expressed concern about promoting effective social participation.

Fig. 5.

Fig. 5

Open in a new tab

Stakeholder consolidated priorities in Niteroi city based on pairwise comparison (AHP).

University experts consider that guaranteed financial resources and increasing the investments in education are the most important criteria for promoting sustainable cities. Improving the quality of education, planning, and stimulating social control were also identified as priorities among the universities. The business sector showed strong preference for increasing social participation, especially throughout dialogue with stakeholders. Promoting alternatives to integrated economic and social development, especially in poor communities, and increasing efficiency of public services were also ranked as important criteria by the business sector. The access to basic services was identified as a criterion but was given less importance. State audit office representatives gave the highest importance to integration between systems for control purposes, but also, enhancing transparency and strengthen social control, as well as increasing mechanisms for communication with citizens.

Regarding the CIVAP region, Fig. 6 shows the preference of municipal government and scientific and technological community representatives. For the municipal government representatives, compliance with the law is the most important criteria for promoting sustainability in the region. The quality of waste management was also identified as an important criteria as well as social participation and institutional aspects related to local public service.

Fig. 6.

Fig. 6

Open in a new tab

Stakeholder consolidated priorities in CIVAP region based on pairwise comparison (AHP).

Scientific and technological community representatives consider that efficient public policies and educational campaigns are the most important criteria to SDG 11 implementation in the region. Also, enhancing social participation and establish partnerships were identified as important criteria.

2.3. Discussion

Overall, the results provided in this research reflected the major urban issues in Niteroi city and CIVAP region. The results also brought to light important findings regarding key stakeholder's opinions on SDG 11 achievement. The results show that no local system developed to evaluate sustainable cities in Brazil is totally adequate, suggesting shifts in the ongoing model. Perhaps, municipal government should better integrate planning tools and SDG 11 indicators. Also, the disclosure of municipal data was an issue in this research. Aspects related to governance and social participation need to be taken into consideration; as well, qualitative indicators should be added to the systems of evaluation. Adding some qualitative indicators could provide valuable information to assessments based only on quantitative indicators (Keirstead, 2007). In practical terms, some adjustment of existing indicators will be necessary - mainly disaggregation through gender, age, and persons with disabilities, and integration between plans and policies to gain better information and decision-making. In this sense, special attention should be given to the lack of integration between climate change and disaster risk reduction.

Regarding the process of selecting indicators, although many existing approaches have been developed, creating indicators is still a challenging task (Lützkendorf and Balouktsi, 2017) and can fail to attain the desired performance if an inadequate process is adopted (Organisation for Economic Co-operation and Development, 2003; Pupphachai and Zuidema, 2017). Also, the selection of indicators cannot be based only on gathering information but also on facilitating the analysis of progress of critical issues (Shen et al., 2011). The methodology adopted in this research can be helpful for facilitating stakeholder participation at local levels, including a comprehensive evaluation of stakeholders' concerns, and facilitating consensus-building (Macharis et al., 2009; Macharis et al., 2012). The methodology allows a better understanding about distinct stakeholders' expectations because it includes all points of view and several criteria in the analysis, avoiding the situation where the results are the least of the outcomes (Innes, 2004). The adoption of a localized approach to select indicators can give a more accurate picture about urban dynamics (Gómez-Álvarez et al., 2018) and identify key issues that demand transformations in Niteroi city and CIVAP region. Importantly, the methodology can be used by all municipalities and also other governmental levels since it is not exclusively applied.

Hence, MAMCA results can be seen from four perspectives: municipal government, universities/scientific and technological community, business sector, and audit state office. In Niteroi city, the municipal government representatives considered that the indicators are helpful for improving public services and reducing some important urban vulnerabilities. For the universities, the indicators can be used as tools for better public management and for enhancing social participation. For the business sector, the indicators are useful for improving services for the public and improving social participation. The audit state office considers that indicators can help to enhance local governance, mainly matters of transparency and social control. In CIVAP region the indicators chosen by the municipal government can help to improve social participation but mainly institutional aspects. For the scientific and technological community, the indicators should be using to enhance efficiency of public policies and quality of education. Indeed, evaluations involving different geographical levels can lead to distinct perspectives and different outcomes (Grafakos et al., 2015). Significantly, to the best of my knowledge, although not conclusive, those results are relevant because the analysis of SDG 11 based on MAMCA has not been previously performed in Brazilian municipalities or other cities.

Based on the results, it is also important to acknowledge the interlinkages and interdependencies among goals to guarantee an integrated implementation of SDGs. In this study, key stakeholders focused on the need for integrating SDG 11 targets with other goals: SDG 4 (Quality education), SDG 8 (Inclusive and sustainable economic growth), SDG 16 (Peace, justice, and strong institutions), and SDG 17 (Partnerships and means of implementation). Significantly, integration among the means of implementation themselves has been highlighted as a factor of success of Agenda 2030 (Stafford-Smith et al., 2017).

In terms of limitations, there are some concerns to take into account. First, most of data required to measure progress towards SDG 11 will have to be generated through municipal government, which is not well structured for data collection and analysis. Second, the narrow representability of the business sector and the lack of NGO participation were key limitations in this research. Furthermore, the difficulty in promoting social participation at local levels, and the lack of systematic data collection and reporting by Brazilian municipalities, have been reported as barriers for successful achievement of goal 11 (Almeida et al., 2018). Considering the difficulties associated with SDG reporting when it is coordinated at the national level (Patel et al., 2017), the indicators identified in this research can be helpful for informing progress to the national level. Moreover, the available indicators have great potential to evaluate the achievement of SDG 11 targets and can be applied to different types of cities.

3. Conclusions

This research concludes that the adopted methodology can support local decision-makers to build a more effective implementation of SDG 11 in Brazilian municipalities by mainly providing an overview of major urban issues based on a multi-stakeholder view. The definition of alternatives to be analysed (step 1) and the selection of stakeholders (step 2) are critical steps in the methodology to legitimise the outcomes. The selection of key stakeholders was a limitation in this study. Regarding the choices and the weights of the criteria (step 3), the technique adopted in this research was suitable but other approaches could be tested. Overall, the MAMCA facilitated communication and deeper analysis by distinct stakeholders, indicating the most important issues regarding SDG 11 implementation. Also, through the methodology it was possible to identify convergences and conflicts between key stakeholders, which should be considered in the further development of local policies and decision making.

The identification of criteria, indicators and measurement methods (step 4) is very useful and can facilitate implementation and evaluation of local policies. The developed indicators can also be used as a starting point to assess the contribution of local policies already deployed in the city to SDG 11 achievement. Also, the indicators can be used to build the municipal baseline and contribute to national achievements and reporting. Therefore, the use of indicators based on a multi-stakeholder view can support the municipal government in identifying new opportunities for sustainable urban development, enabling more systemic and holistic views to facilitate formulation of local policies.

The Analytical hierarchy process (AHP) used to develop step 5 was very suitable and facilitated the analysis of different stakeholders' points of view. Regarding the outcomes (step 6 and 7), although the results are insightful, considering that a broad participation was not possible to achieve, a validation phase including new local stakeholders' views should be incorporated for further refinement and improvement of indicators. In this phase, more indicators could be added or eliminated. Significantly, considering the distinct realities in the country, it's likely that the set of indicators developed in this research would not suit all municipalities, requiring some adjustments.

Finally, the results revealed that existing systems of indicators will not be able to demonstrate progress towards achieving SDG 11 targets at local levels. It is necessary to adjust existing indicators and organize a consistent and frequent method to gather and evaluate data as well as mobilize civil society towards the importance of SDGs' achievements. Moreover, data and the updated frequency that facilitates to monitor evolutions.

Declarations

Author contribution statement

Ana C. L. Almeida: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.

Acknowledgements

I would like to acknowledge and thank all participants in Niteroi city and CIVAP for their valuable and significant contribution to this work.

References

  1. Alkhalidi A., Qoaider L., Khashman A., Al-Alami A.R., Jiryes S. Energy and water as indicators for sustainable city site selection and design in Jordan using smart grid. Sustain. Cities Soc. 2018;37:125–132. [Google Scholar]
  2. Almeida A.C.L., Smart J.C.R., Davey P. Can learned experiences accelerate the implementation of sustainable development goal 11? A framework to evaluate the contributions of local sustainable initiatives to delivery SDG 11 in Brazilian municipalities. Eur. J. Sustain. Dev. 2018;7(4):517–530. [Google Scholar]
  3. Berg B.L. Allyn and Bacon; 2001. Qualitative Research Methods for the Social Sciences. [Google Scholar]
  4. Bergqvist R., Macharis C., Meers D., Woxenius J. Making hinterland transport more sustainable a multi actor multi criteria analysis. Res. Transport. Bus. Manag. 2015;14:80–89. [Google Scholar]
  5. Braulio-Gonzalo M., Bovea M.D., Ruá M.J. Sustainability on the urban scale: proposal of a structure of indicators for the Spanish context. Environ. Impact Assess. Rev. 2015;53:16–30. [Google Scholar]
  6. Confederação Nacional de Municípios . 2017. SDG Mandala.http://ods.cnm.org.br/mandala-municipal Retrieved from. [Google Scholar]
  7. Costa M.d.S., Ramos R.A., Silva A.N. R.d. 2007. Índice de mobilidade urbana sustentável para cidades brasileiras. [Google Scholar]
  8. De Brucker K., Macharis C., Verbeke A. Multi-criteria analysis and the resolution of sustainable development dilemmas: a stakeholder management approach. Eur. J. Oper. Res. 2013;224(1):122–131. [Google Scholar]
  9. European Commission DG Environment . 2018. Science for Environment Policy. Indicators for Sustainable Cities. In-depth Report 12. [Google Scholar]
  10. Florissi E. 2009. Desenvolvimento urbano sustentável: um estudo sobre sistemas de indicadores de sustentabilidade urbana. [Google Scholar]
  11. Gomes A.M., Reys P.d.S. 2015. Niterói que queremos: uma estratégia inovadora de planejamento. [Google Scholar]
  12. Gómez-Álvarez D., López-Moreno E., Bilsky E., Ochoa K.B., Osorio E. Indicators for measuring urban sustainability and resilience. In: Elmqvist T., Bai X., Frantzeskaki N., Maddox D., editors. The Urban Planet: Knowledge towards Sustainable Cities. Cambridge University Press; 2018. pp. 163–179. [Google Scholar]
  13. Grafakos S., Flamos A., Enseñado E. Preferences matter: a constructive approach to incorporating local stakeholders’ preferences in the sustainability evaluation of energy technologies. Sustainability. 2015;7(8):10922–10960. [Google Scholar]
  14. Greco S., Figueira J., Ehrgott M. Springer; 2016. Multiple Criteria Decision Analysis. [Google Scholar]
  15. Hák T., Janoušková S., Moldan B. Sustainable Development Goals: a need for relevant indicators. Ecol. Indicat. 2016;60:565–573. [Google Scholar]
  16. Hiremath R.B., Balachandra P., Kumar B., Bansode S.S., Murali J. Indicator-based urban sustainability - a review. Energy Sustain. Dev. 2013;17(6):555–563. [Google Scholar]
  17. Innes J.E. Consensus building: clarifications for the critics. Plan. Theory. 2004;3(1):5–20. [Google Scholar]
  18. Instituto Brasileiro de Geografia e Estatistica . 2015. Pesquisa Nacional por Amostra de Domicílios – PNAD.https://www.ibge.gov.br/estatisticas-novoportal/sociais/educacao/9127-pesquisa-nacional-por-amostra-de-domicilios.html?=&t=destaques Retrieved 08/03/2018. [Google Scholar]
  19. Instituto Brasileiro de Geografia e Estatistica . 2017. Estimativas de População.https://www.ibge.gov.br/estatisticas-novoportal/sociais/populacao/9103-estimativas-de-populacao.html?=&t=destaques Retrieved 08/03/2018. [Google Scholar]
  20. Instituto Brasileiro de Geografia e Estatistica . 2018. Objetivos de Desenvolvimento Sustentavel.https://indicadoresods.ibge.gov.br/objetivo/objetivo?n=11 Retrieved 23/05/2019. [Google Scholar]
  21. Keirstead J. Paper Presented at the International Conference on Whole Life Urban Sustainability and its Assessment Glasgow. 2007. Selecting sustainability indicators for urban energy systems. [Google Scholar]
  22. Kourtit K., Macharis C., Nijkamp P. A multi-actor multi-criteria analysis of the performance of global cities. Appl. Geogr. 2014;49:24–36. [Google Scholar]
  23. Lützkendorf T., Balouktsi M. Assessing a sustainable urban development: typology of indicators and sources of information. Proc. Environ. Sci. 2017;38:546–553. [Google Scholar]
  24. Macharis C., De Witte A., Ampe J. The multi-actor, multi-criteria analysis methodology (MAMCA) for the evaluation of transport projects: theory and practice. J. Adv. Transp. 2009;43(2):183–202. [Google Scholar]
  25. Macharis C., De Witte A., Turcksin L. The Multi-Actor Multi-Criteria Analysis (MAMCA) application in the Flemish long-term decision making process on mobility and logistics. Transp. Policy. 2010;17(5):303–311. [Google Scholar]
  26. Macharis C., Turcksin L., Lebeau K. Multi actor multi criteria analysis (MAMCA) as a tool to support sustainable decisions: state of use. Decis. Support Syst. 2012;54(1):610–620. [Google Scholar]
  27. Macharis C., Verbeke A., De Brucker K. The strategic evaluation of new technologies through multicriteria analysis: the ADVISORS case. Res. Transp. Econ. 2004;8:443–462. [Google Scholar]
  28. Munier N. Methodology to select a set of urban sustainability indicators to measure the state of the city, and performance assessment. Ecol. Indicat. 2011;11(5):1020–1026. [Google Scholar]
  29. Organisation for Economic Co-operation and Development . 2003. OECD Environmental Indicators: Development, Measurement and Use. [Google Scholar]
  30. Patel Z., Greyling S., Simon D., Arfvidsson H., Moodley N., Primo N., Wright C. Local responses to global sustainability agendas: learning from experimenting with the urban sustainable development goal in Cape Town. Sustain. Sci. 2017;12(5):785–797. doi: 10.1007/s11625-017-0500-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Prefeitura de Niterói . In: Niterói Que Queremos Ter - Plano Estratégico 2013-2033. Niterói P.d., editor. 2017. [Google Scholar]
  32. Programa Cidades Sustentaveis . 2017. Anexo - Guia Gestao Publica Sustentavel: Indicadores do Programa Cidades Sustentáveis e Orientações para o Plano de Metas. [Google Scholar]
  33. Pupphachai U., Zuidema C. Sustainability indicators: a tool to generate learning and adaptation in sustainable urban development. Ecol. Indicat. 2017;72:784–793. [Google Scholar]
  34. Saaty T.L. RWS Publications; USA: 1980. The Analytical Hierarchy Process, Planning, Priority. Resource Allocation. [Google Scholar]
  35. Science for Environment Policy . 2018. Indicators for Sustainable Cities.http://ec.europa.eu/environment/integration/research/newsalert/pdf/indicators_for_sustainable_cities_IR12_en.pdf In-depth Report 12. Retrieved from. [Google Scholar]
  36. Shen L.-Y., Ochoa J.J., Shah M.N., Zhang X. The application of urban sustainability indicators–A comparison between various practices. Habitat Int. 2011;35(1):17–29. [Google Scholar]
  37. Stafford-Smith M., Griggs D., Gaffney O., Ullah F., Reyers B., Kanie N. Integration: the key to implementing the sustainable development goals. Sustain. Sci. 2017;12(6):911–919. doi: 10.1007/s11625-016-0383-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Stathopoulos A., Valeri E., Marcucci E. Stakeholder reactions to urban freight policy innovation. J. Transp. Geogr. 2012;22:34–45. [Google Scholar]
  39. Sun H., Zhang Y., Wang Y., Li L., Sheng Y. A social stakeholder support assessment of low-carbon transport policy based on multi-actor multi-criteria analysis: the case of Tianjin. Transp. Policy. 2015;41:103–116. [Google Scholar]
  40. Tran L. An interactive method to select a set of sustainable urban development indicators. Ecol. Indicat. 2016;61:418–427. [Google Scholar]
  41. Tribunal de Contas do Estado de Sao Paulo . Sao Paulo; 2014. Indice de efetividade da gestao municipal. [Google Scholar]
  42. Turcksin L., Macharis C., Lebeau K., Boureima F., Van Mierlo J., Bram S. A multi-actor multi-criteria framework to assess the stakeholder support for different biofuel options: the case of Belgium. Energy Pol. 2011;39(1):200–214. [Google Scholar]
  43. United Nations . 2015. Framework Convention on Climate Change. FCC/CP/2015/L.9. [Google Scholar]
  44. United Nations Development Programme, UN-Habitat, Global Taskforce of Local and Regional Governments . 2016. Roadmap for Localizing the SDGs: Implementation and Monitoring at Subnational Level. [Google Scholar]
  45. United Nations General Assembly . 2017. Work of the Statistical Commission Pertaining to the 2030 Agenda for Sustainable Development. United Nations RES/A/71/313. [Google Scholar]
  46. Xiang N. Paper Presented at the IOP Conference Series: Earth and Environmental Science. 2017. A Review on the research and practice of city sustainable development indicators and indices. [Google Scholar]
  47. Yang B., Xu T., Shi L. Analysis on sustainable urban development levels and trends in China's cities. J. Clean. Prod. 2017;141:868–880. [Google Scholar]
  48. Yin R.K. Case study research design and methods third edition. Appl. Soc. Res. Meth. Ser. 2003;5 [Google Scholar]

Articles from Heliyon are provided here courtesy of Elsevier

RESOURCES