Abstract
The research in sustainable development goals (SDG) increases year by year since its approval in 2015. Typically, after a phase of exponential growth, the number of publications increases at lower rates, suggesting a consolidation process in which literature reviews become a relevant and high-evidence type of document. In this context, the aim of this study was to perform an unprecedented bibliometric analysis of literature reviews on SDG to assess the evolution and consolidation of the scientific research. Article reviews on SDG from 2015 to 2022 were retrieved from Web of Science core collection and a descriptive bibliometric analysis was performed by growth rate, research area, source, citation, and region. Mapping and cluster analysis using keyword co-occurrence, co-authorship, and bibliographic coupling were also applied. The result revealed that SDG is a fast-growing field, with a trend in the diversification of research areas. Most of the review documents were categorized in general aspects of sustainability. Technology (SDG 9) and economic growth (SDG 8) were spotted as hidden key research areas. This result is contrary to previous bibliometric studies on SDG, demonstrating the rapid evolution and change in the field. In addition, literature reviews on reduced inequalities (SDG 10), gender equality (SDG 5); oceans, seas, and marine environments (SDG 14); and peace, justice, and strong institutions (SDG 16) were revealed as research gaps. Thus, the results demonstrated that the research on SDG cannot yet be considered a consolidated area of research, as it leaves many SDG unexplored. Future research has been proposed accordingly.
Keywords: Agenda 2030, Citation analysis, Health promotion, Scientometrics, SDG, Sustainability
Introduction
The United Nations (UN) sustainable development goals (SDG) are a universal political agenda that address for a collective action to achieve a better and more sustainable future for all, solving the social, economic, and environmental issues that hinder global progress towards sustainability intended to be achieved by the year 2030 (United Nations 2015).
The UN General Assembly approved the Resolution A/RES/70/1 on “Transforming our world: the 2030 Agenda for Sustainable Development.” The agenda outlined 17 SDG and specific targets and indicators for each of the 17 SDG were defined by UN, totalizing 169 targets and 213 indicators that form a global action plan (United Nations 2017). Furthermore, the Agenda established five areas of critical importance known as the five pillars (5Ps): people, planet, prosperity, peace, and partnership (Tremblay et al. 2020) (Table 1).
Table 1.
Goal | Title | # Targets | # Indicators | 5Ps* classification |
---|---|---|---|---|
1 | No poverty | 7 | 13 | People |
2 | Zero hunger | 8 | 13 | People |
3 | Good health and well-being | 13 | 28 | People |
4 | Quality education | 10 | 12 | People |
5 | Gender equality | 9 | 14 | People |
6 | Clean water and sanitation | 8 | 11 | People, planet |
7 | Affordable and clean energy | 5 | 6 | Prosperity, planet |
8 | Decent work and economic growth | 12 | 16 | Prosperity, people |
9 | Industry, innovation and infrastructure | 8 | 12 | Prosperity |
10 | Reducing inequalities | 10 | 14 | Prosperity, people |
11 | Sustainable cities and communities | 10 | 15 | Prosperity |
12 | Responsible consumption and production | 11 | 13 | Planet |
13 | Climate action | 5 | 8 | Planet |
14 | Life below water | 10 | 10 | Planet |
15 | Life on land | 12 | 14 | Planet |
16 | Peace, justice, and strong institutions | 12 | 14 | Peace |
17 | Partnerships for the goals | 16 | 24 | Partnership |
*5Ps, planet, prosperity, people, peace, and partnership
The SDG are a recognized blueprint essential to achieve shared and sustainable prosperity with global action among governmental and non-governmental organizations, businesses, industry, civil society organizations, research, and technology development (Khaled et al. 2021).
However, there are crucial challenges to overcome, emphasizing the importance of the interrelationship between sectors, actors, and countries that have lesser and greater economic development (Stafford-Smith et al. 2017). Additionally, it can be pointed out the strong interdependencies between the failure or delay to implement one goal and how it will have repercussion in the others goals (Randers et al. 2018; Díaz-López et al. 2021).
Consequently, due to these interrelationships complexity, it is valuable for researchers to assess the status of the SDG research, for instance by mapping the existing knowledge or creating new knowledge to contribute to achieve the goals defined by the United Nations and also allow the overcoming of previous partial approaches to sustainable development (Belmonte-Ureña et al. 2021; Bordignon 2021).
A simple query performed through Web of Science (WoS) using the keywords “Sustainable Development Goal*” performed on October 1, 2022 resulted in 37,937 records. This demonstrates the great interest in the SDG as an object of research in recent years. Given these numbers, it is relatively difficult to map and identify the status of SDG research because of their infinity. Other issues deal with subjectivity, transparency, and delay in the literature review process. A broad view of a research area is important for obtaining valuable and impartial prospects for future research developments. Thus, a comprehensive review is needed to facilitate the integration of the contributions to provide a critical perspective (Díaz-López et al. 2021).
Bibliometric analysis is a statistical technique applied to examine the scientific production in a field of research. It allows to study the evolution of knowledge on a given topic during a certain period of time based on data publication (Belmonte-Ureña et al. 2021; Zupic and Čater 2015). It combines two main procedures: (i) the performance analysis and (ii) science mapping. The performance analysis is established on indicators that provide data about the amount and impact of the research through the application of several techniques, as citation analysis, counting publications, word frequency analysis by a unit of analysis (Romanelli et al. 2021). Science mapping is a graphic representation of how different scientific elements (knowledge areas, documents or authors) are interrelated. It shows the impact, structural, and dynamic organization of a knowledge topic, a field of research, a group of researchers, or a document, based on relation indicators (Marzi et al. 2017; Pizzi et al. 2020). Science mapping allows finding insights into patterns of a knowledge area that would be difficult to identify using traditional research review methods (Hallinger and Chatpinyakoop 2019; Prieto-Jiménez et al. 2021). Furthermore, science mapping analysis can be used to show or uncover some invisible key elements in a specific interest area (Cobo et al. 2011).
Although many bibliometric studies have been found in the field of SDGs research, most of them focus on specific disciplines, such as SDG and the business sector (Pizzi et al. 2020), education (Prieto-Jiménez et al. 2021), poverty (Yu and Huang 2021), and few bibliometric studies cover the SDG general aspects and its evolution. Nevertheless, some interesting bibliometric studies of SDG trends are worth mentioning, such as the study of Díaz-López et al. (2021), Meschede (2020), Yeh et al. (2022), and Sianes et al. (2022). All of them included a large number of documents (thousands) and do none of them focused on literature review articles.
Thus, the aim of this study was to conduct a bibliometric analysis of literature reviews on SDG from 2015 to 2022 with the following specific objectives:
Get a perspective of the status and evolution on the scientific research of SDG
Provide a visual representation of interrelations of the SDG review articles and its scientific elements
Reveal insights from the identified patterns of thematic currents
Define research gaps and hidden key elements on the SDG and propose future research
Materials and methods
The analysis of scientific literature was performed using a bibliometric analysis and was conducted in three phases (Fig. 1). It is worth mentioning that the present study restricted the analysis to review articles, that capture a general view of SDG research, and allow to understand and identify the domain of knowledge, the development of theories and concepts, and the academic debates in SDG research without carrying out any new studies or exhaustive review of the literature, as the review articles examine and summarize the state-of-art on certain topics from the available literature (Meschede 2020).
Search and data collection
Analyzed metadata used in this investigation were obtained from the Clarivate Analytics WoS core collection database of the Institute for Scientific Information (ISI, Philadelphia, PA). Documents were retrieved by searching (‘‘SUSTAINABLE DEVELOPMENT GOAL*’’ or ‘‘SDG*’’) in the field “TITLE”, on SCI-expanded collection and as filter “REVIEW ARTICLES.” The search was conducted on October 1, 2022 and was narrowed to documents with publications years after 2015, because of the adoption year of the SDG, resulting in 312 documents.
All available metadata (abstract, keywords, funding, author, authors’ affiliation information, year of publication, thematic area, journal) were downloaded as a CSV-file. The data were checked for debugging using Microsoft Excel software, and a thesaurus file was created. In the thesaurus file, the keywords were normalized, eliminating duplicities, unifying synonyms, and developing acronyms. For this purpose, the all keywords were included. The VOSviewer 1.6.18 software was selected for this phase, due to its remarkable visualization feature for bibliometric data and also because it is a freely available tool (Meschede 2020; Prieto-Jiménez et al. 2021; van Eck and Waltman 2010).
Performance analysis
In this step, the basic characteristics of retrieved documents was performed using a descriptive bibliometric analysis exploring: (1) publication output; (2) research area; (3) most productive sources; (4) most-cited documents; (5) most productive countries. For the analysis of the most relevant sources, the impact factors were obtained from the Journal Citation Reports (JCR) published in 2021 that assesses the journals performance via the SCIMago Journal Rank (SJR) indicator based on an average number of citations. Furthermore, the total citations (TC), the average number of citations per paper (AC), the normalized citations (NC), the average normalized citations (ANC), and Hirsch index (h-index) were also used to assess the citation impact and productivity of a document, authors, and/or sources.
Cluster analysis and visualization
The third step included the cluster analysis and visualization by mapping technique. The co-authorship, bibliographic coupling and keyword co-occurrence were selected to be used as indicators (Ferreira 2018; Santana et al. 2020). The maps were interpreted according to the generated weights and score attributes assigned to each cluster (Garrigos-Simon et al. 2018; Prieto-Jiménez et al. 2021). An interpretive analysis was used to explore the conceptual structure of the SDG field and identify the thematic currents. Unique and significant keywords were identified in the co-occurrence mapping analysis. Thus, transversal terms (e.g., “sustainability”, “sustainable development goals” or “agenda 2030”) were excluded, as they can be associated with several SDG or the entire sustainable development agenda. In addition, keywords relating to specific methodologies (e.g., “review”, “bibliometric analysis”, “cluster analysis” or “systematic review”) were also excluded according to previous reported methodologies (Yakovleva and Vazquez-Brust 2012; Pizzi et al. 2020; Belmonte-Ureña et al. 2021).
Results and discussion
Publication output
Figure 2 shows the cumulative distribution of scientific production of documents output of SDG reviews from its first issue in 2015 to 2022, after 7 years of its conception. In general, the publication output seems to increase over the years. In 2015, only one review was published, related to SDG 3, and the number of published review papers increased to 9 in the year of 2016. From 2017 to 2018, the number of published review papers per year increased dramatically from 10 articles to 40. Three years later, the number of published papers per year doubled and from then onwards, continues to grow, achieving a total of 312 publications in October 2022.
Research areas
The cumulative publications of the main fields covered by literature reviews of SDG classified by WoS thematic categories published over the years are also shown in Fig. 2. The main fields covered were (i) environmental sciences, (ii) green sustainable science technology, (iii) environmental studies, (iv) public environmental occupational health, (v) water resources, and (vi) energy fuels. The first category represents 44% of the total documents analyzed and it is of particular relevance. This result was expected, as environmental sciences is an interdisciplinary academic field that integrates physics, biology, and geography that studies the environment and the solution of environmental issues and includes a wide range of subjects (Zhu et al. 2021). Also, the publications could be included in multiple categories. Most of the documents (56%) were categorized in general aspects of sustainability, such as environmental sciences, environmental studies, and green and sustainable science technology. The others thematic categories that stood out were attributed to the relevance of SDG 3 and to technological themes that include SDG 6 and SDG 7, respectively.
In order to assess the distribution of SDG in literature review publications over the years, the articles were analyzed individually and classified as shown in Fig. 3. An initial concern with SDG 3 was observed in relation to the first few years, since the first single publication in 2015, was referred to public environmental occupational health, and consequently with 100% of publications. A decrease trend in the proportion of SDG 3 publications over years was observed. It is clear that publications related to SDG 3 increased in number; however, the proportion of articles concerned to SDG 3 decreased, while the number and proportion of other SDG increased.
In the second year, an interest in relation to SDG 2, 12, and 17 was observed. Only in 2017 review articles of SDG related to hunger, energy, and peace were found. From 2018 onwards, more applied and diversified research were observed. Thus, areas related to nature and technology were covered, as water and sanitation, industry, innovation and infrastructure, food and agriculture, business and management, development studies and urban studies, climate change, life on land and water, are also slowly growing, indicating a trend in research in the environmental area. In addition, the results showed that the social area was the least studied, since only 2 and 4 review articles were found with regard to SDG 10 and SDG 5, respectively. The SDG are integrated, indivisible, and mixed, in a balanced way, in the three dimensions of sustainable development: economic, social, and environmental. Therefore, it will not be possible to achieve the goals if there is no balance between them (United Nations 2015).
Furthermore, it was observed that even though the year 2022 has not yet ended, this was the only year that presented review articles that includes all the SDG. This result also shows the trend of more diversified research on SDG.
Sources
The most relevant journals regarding the largest number of citations are presented in Table 2. Among the top ten journals, most of them are related to general aspects of sustainability, such as, the journals: Current Opinion in Environmental Sustainability, Science of the Total Environment, Journal of Cleaner Production, Sustainability and Sustainability Science. Most journals are in regard to the general aspects of sustainability, according to the previous results. The journal Lancet and Globalization and Health are related to SDG 3, whereas Nature energy and Renewable Sustainable Energy Reviews are attributed to SDG 7. These results are in accordance to the results previously discussed.
Table 2.
No. | Journal | D | TC | AC | NC | ANC | IF | h-index | Publisher |
---|---|---|---|---|---|---|---|---|---|
1 | Current Opinion in Environmental Sustainability | 12 | 790 | 66 | 12 | 1.02 | 6.984 | 99 | Elsevier |
2 | Lancet | 4 | 441 | 110 | 11 | 2.77 | 79.321 | 807 | Elsevier |
3 | Science of the Total Environment | 8 | 479 | 60 | 27 | 3.35 | 7.963 | 275 | Elsevier |
4 | Journal of Cleaner Production | 13 | 532 | 41 | 21 | 1.62 | 9.297 | 232 | Elsevier |
5 | Sustainability | 40 | 492 | 12 | 27 | 0.67 | 3.251 | 109 | MDPI |
6 | Nature Energy | 1 | 372 | 372 | 6 | 6.03 | 60.85 | 160 | Springer Nature |
7 | Renewable Sustainable Energy Reviews | 6 | 270 | 45 | 5 | 0.78 | 14.982 | 337 | Elsevier |
8 | Nature Communications | 2 | 294 | 147 | 11 | 5.43 | 14.919 | 410 | Springer Nature |
9 | Sustainability Science | 7 | 240 | 34 | 7 | 1.01 | 6.367 | 65 | Springer |
10 | Globalization and Health | 4 | 193 | 48 | 2 | 0.48 | 4.185 | 61 | Springer Nature |
D, number of documents; TC, total citations; AC, average number of citations per document; NC, normalized citations; ANC, average normalized citations; h-index, Hirsch index; IF, impact factor
Regarding the number of documents by journal, Sustainability has the largest number of reviews on SDG, concentrating 40 (13.8%) of all the articles analyzed (312), followed by the Journal of Cleaner Production with 4.2% participation of the review articles (13), and Current Opinion in Environmental Sustainability journal with 3.8% of the total dataset, that is also the most-cited journal with 790 total citations (TC). Concerning the impact factor, two journals stand out, the 79-point journal Lancet and the 61-point journal Nature Energy. In respect to the h-index, the Lancet (807) and Nature Communications (410) are the most remarkable. Similar results were obtained by Sianes et al. (2022) who used bibliometric methodologies to evaluate the impacts of SDG on the academic agenda.
The average number of citations per document (AC) show that Nature Energy, Nature Communications, and the Lancet stand out with 372, 147, and 110 citations, respectively. However, when observing the normalized citation (NC), that is the number of citations of a document equals the number of citations of the document divided by the average number of citations of all documents published in the same year included in the data that is provided to VOSviewer (van Eck and Waltman 2010), showed that Science of the Total Environment (NC = 27), Sustainability (NC = 27), and the Journal of cleaner production (NC = 21) are the most prominent sources. This normalization corrects the fact that older documents have more time to receive citations than recent documents (van Eck and Waltman 2010). Nonetheless, the average normalized citation (ANC), which indicates the NC divided by the number of documents, demonstrated that Nature Energies (ANC = 6.03), Nature Communications (ANC = 5.43), and the Science of total environment (ANC = 3.35) are the most noticeable journals.
Thus, it is noted that even though the total number of documents, TC, h-index, and IF indicate that SDG 3 is among the top 3 most relevant sources, the normalized citation data indicate that journals research on energy, multidisciplinary and general environmental sciences, and sustainability are among the most noticeable sources. It is important to highlight that this result is contrary to that obtained in bibliometric studies reported previously (Meschede 2020; Yeh et al. 2022). This indicates that the number of citations alone is not adequate to assess the relevance of a document or journal (Simko 2015).
Documents
One can observe in Table 3, which show the most relevant review documents based on the TC of the documents, that the majority are focused on SDGs 3 and 7. In this way, review publications go even further in determining strategies and methodologies that seek to achieve the SDG through serving the health and energy sectors. However, when analyzing the NC, the document that really stands out is related to SDG 9, followed by the SDGs 7 and 3, which corresponds to innovation, energy and mental health, respectively.
Table 3.
No. | Title | Source title | Related SDG | TC | NC | Ref |
---|---|---|---|---|---|---|
1 | Mapping synergies and trade-offs between energy and the Sustainable Development Goals | Nature Energy | 7 | 372 | 6.03 | Nerini et al. (2018) |
2 | The role of artificial intelligence in achieving the Sustainable Development Goals | Nature Communications | 9 | 286 | 10.01 | Vinuesa et al. (2020) |
3 | Global governance by goal-setting: the novel approach of the UN Sustainable Development Goals | Current Opinion in Environmental Sustainability | 16 | 283 | 3.17 | Biermann et al. (2017) |
4 | Social determinants of mental disorders and the Sustainable Development Goals: a systematic review of reviews | Lancet Psychiatry | 3 | 259 | 4.20 | Lund et al. (2018) |
5 | Health in the sustainable development goals: ready for a paradigm shift? | Globalization and Health | 3 | 181 | 1.00 | Buse and Hawkes (2015) |
6 | A literature-based review on potentials and constraints in the implementation of the sustainable development goals | Journal of Cleaner Production | 17 | 166 | 2.69 | Caiado et al. (2018) |
7 | Investing in non-communicable disease prevention and management to advance the Sustainable Development Goals | Lancet | 3 | 153 | 2.48 | Nugent et al. (2018) |
8 | Connecting the sustainable development goals by their energy inter-linkages | Environmental Research Letters | 7 | 152 | 2.46 | McCollum et al. (2018) |
9 | Advancing global health and strengthening the HIV response in the era of the Sustainable Development Goals: the International AIDS Society-Lancet Commission | Lancet | 3 | 149 | 2.41 | Bekker et al. (2018) |
10 | Initial progress in implementing the Sustainable Development Goals (SDGs): a review of evidence from countries | Sustainability Science | 17 | 148 | 2.40 | (Allen et al. 2018) |
10 | Physical water scarcity metrics for monitoring progress towards SDG target 6.4: An evaluation of indicator 6.4.2 “Level of water stress” | Science of The Total Environment | 6 | 148 | 2.40 | (Vanham et al. 2018) |
SDG, sustainable development goals; TC, total citations; NC, normalized citations
Furthermore, the density analysis of bibliographic coupling of the most-cited documents is represented in Fig. 4. For this purpose, a minimum of 148 citations from each document were used as a limit, in order to obtain the top 10 most-cited documents. Thus, 11 documents met the stipulated threshold, as expected, and 4 clusters were formed with 21 links, and total link strength of 56, where the weights were the number of citations. In general, the documents did not present high coupling strengths, as the number of citations is represented by the intensity of the red color and the bibliographic coupling by the proximity of the documents. This behavior was expected, as those papers have distinct themes and their specific attention contribute to SDG and novelty on different areas.
Nerini et al. (2018) mapped the synergies and trade-offs in energy and SDG, and Vinuesa et al. (2020) described about the importance of artificial intelligence. Both documents discussed about technology and SDG, and, thus, they present some coupling strength. However, McCollum et al. (2018) also discussed about energy (SDG 7) and did not present bibliographic coupling with those articles. These results of bibliographic coupling demonstrated the lack of integration between the research, even when within the same field.
Caiado et al. (2018) and Biermann et al. (2017) discuss similar themes of constraints and governance. Therefore, it was expected them to have high coupling strengths. Vanham et al. (2018) was the only article about the SDG 6 included in the most-cited reviews, which contains water scarcity metrics for monitoring progress, and thus, it was not found bibliographic coupling with others top-cited documents.
Buse and Hawkes (2015), Bekker et al. (2018), and Nugent et al. (2018) focused their articles on SDG 3 and reviewed studies about heath, HIV, and disease prevention, respectively. It is worth mentioning that the article of Lund et al. (2018), which brings a review study on social determinants of mental disorders, did not presented bibliographic coupling with the other documents of SDG 3. This is an indicative that the psychiatry field is traditionally separated from other medicine branches of medicine and health fields (Fiorillo and Maj, 2018), even though mental health should be considered as good health and well-being (SDG 3). These results demonstrate that the areas are not very interconnected even when dealing with the same subject.
Countries
In order to represent the importance of SDG review production by the most productive countries based on corresponding authors affiliations country, a total of 88 countries, from all continents, of the place of the scientific production analyzed by country or region were represented (Fig. 5). The United Kingdom was the most prolific region, accounting for 21.7% of the review articles (68), followed by the USA and Australia, which represented 19.6% (61) and 16.0% (50), respectively. Finally, India accounted for 10.9% (34) of the scientific production, followed by Germany and China, both accounting with 8% (25) each.
To better understand international collaborations, the mapping of co-authorship of countries according to the authors’ affiliations was presented in Fig. 6. In addition, the citation characteristics of the clusters were provided in Table 4. All countries that presented at least 5 publications were included. This condition was true for 34 countries, and 5 clusters were obtained with 310 links, and total link strength of 758. The size of the circle reflected the number of publications (weights) in the dataset that were associated to the country.
Table 4.
Cluster | # Items | Countries | D | TC | AC | ANC |
---|---|---|---|---|---|---|
1 (blue) | 10 | Singapore, Italy, Austria, Ireland, Sweden, Spain, Germany, Belgium, Netherlands and France | 147 | 8183 | 54.20 ± 17.42 | 1.79 ± 0.43 |
2 (green) | 9 | New Zealand, Kenya, Brazil, Pakistan, China, Canada, Malaysia, Bangladesh, and Thailand | 117 | 4251 | 38.12 ± 30.00 | 1.25 ± 0.63 |
3 (red) | 9 | United Arab Emirates, Denmark, USA, Japan, India, England, Scotland, Norway, and Nepal | 191 | 6979 | 26.06 ± 16.17 | 1.14 ± 0.48 |
4 (yellow) | 4 | South Africa, Mexico, Switzerland, and Nigeria | 58 | 2922 | 48.17 ± 10.17 | 1.22 ± 0.25 |
5 (pink) | 2 | Australia and South Korea | 55 | 2024 | 26.45 ± 17.89 | 0.88 ± 0.55 |
D, number of documents; TC, total citations; AC, average number of citations per year; ANC, average normalized citations
The first cluster was composed practically exclusively of European countries that included Italy, Austria, Ireland, Sweden, Spain, Germany, Belgium, the Netherlands, France, and Singapore. This cluster obtained the higher TC, AC, and ANC, despite it not presented the largest number of documents. An interesting result was that the highest value of ANC was for the only non-European country, Singapore (ANC = 2.48), even with only 5 documents published.
The second cluster consisted of multicontinental cooperation, Oceania, Africa, America, and Asia, composed of the countries, New Zealand, Kenya, Brazil, Pakistan, China, Canada, Malaysia, Bangladesh, and Thailand. The European continent was the only excluded in this cluster. The cluster 2 presented higher ANC when compared to cluster 3, which presented the same number of countries, a greater number of documents and citations, but a smaller percentage of developing countries. This is a strong indicative that developing countries are relevant when it comes to SDG, since there is an exclusive SDG that seeks equality (SDG 10) and this topic remains at the center of global debates (Díaz-López et al. 2021; Bose and Khan 2022).
The third red central cluster involves England and the USA, the most occurring countries. They are linked to almost all countries. However, they have stronger collaboration with United Arab Emirates, Denmark, Japan, India, Scotland, Norway, and Nepal. Despite the USA and England being among the countries with the highest number of documents, this cluster has the AC and ANC among the least cited, indicating that the large number of documents do not reflect the real impact of the review publications related to the SDGs (Bennich et al. 2020).
The fourth cluster comprises South Africa, Mexico, Switzerland, and Nigeria. And again, it shows that documents become more impactful and relevant when there is intercontinental cooperation, as this cluster comprises few countries and documents, but have high AC and ANC. In low- and middle-income countries, limited or non-existent resources are the great challenge for the development of research in all areas of knowledge. On the other hand, the reverse innovation approach (Harris et al. 2020) has increasingly encouraged rich countries to research in poor countries. In this context, low-resource environments become an opportunity for study and learning to share with rich countries, generating promising options for new products, processes, or policies, which, by different contexts, can provide solutions to gaps or unresolved challenges (Bhattacharyya et al. 2017).
Australia was the third most productive country (Fig. 5) and thus, was also linked to almost all analyzed countries. Whereas, Australia is linked especially to South Korea, the fifth cluster entails only two countries with cooperation of 5 documents. Furthermore, South Korea, due to having only link to Australia, presented among the lower ANC. This result is a strong indication that articles produced without internationalization are less relevant (Vrontis and Christofi 2021).
Keywords
The analysis of keywords are essential in reflecting and defining the research contents and can be used to identify the framework of a main research in many areas (Wuni et al. 2019). The most relevant keywords co-occurrence were used to investigate the conceptual structure of the SDG literature reviews to identify the prevailing thematic (Pizzi et al. 2020). Figure 7 and Table 5 present the results of the co-occurrence analysis of the diversity of 1933 of all keywords from 312 documents analyzed with 6 as a minimum number of occurrences of a keyword, and no minimum number of citations and documents. The map resulted in 93 keywords that met the threshold with 6 clusters, 1250 links, and total link strength of 2036. The size of the circle reflected the number of occurrences (weights) of each keyword in the dataset.
Table 5.
Cluster | Most influential keywords* | # Items | ANC | Occurrences | Assigned SDG |
---|---|---|---|---|---|
1 (blue) | Middle-income countries, access, social determinants, community, services, health policy, diseases | 23 | 0.84 ± 0.43 | 10.04 ± 6.44 | 1, 3, 4, 5, 10 |
2 (green) | Industry 4.0, big data, supply chain, nexus, strategy, circular economy, technology, corporate social-reasonability, future, challenges, innovation, energy, impacts, management, business, climate | 21 | 1.67 ± 0.84 | 12.48 ± 8.66 | 7, 8, 9, 12, 13 |
3 (red) | Life-cycle assessment, waste management, tourism, industry, water, environmental impacts, health, nutrition | 17 | 1.03 ± 0.49 | 10.24 ± 8.20 | 3, 6, 8, 9, 11, 12, 13 |
4 (yellow) | Zero hunger, trade-offs, stakeholders, food, land, waste-water, synergy, consumption | 14 | 1.25 ± 0.54 | 10.43 ± 5. 83 | 2, 6, 7, 17 |
5 (pink) | Resources, climate change, land-use, resilience | 12 | 0.80 ± 0.34 | 12.25 ± 13.40 | 13, 15 |
6 (purple) | Electricity, greenhouse-gas emissions | 6 | 1.13 ± 0.73 | 11.00 ± 6.24 | 7, 10, 13 |
ANC, average normalized citation; SDG, sustainable development goals; 5Ps, Planet, prosperity, people, peace and partnership
*Included keywords with ANC > 1.00
Analyzing the corresponding keywords of the review articles in the co-occurrence map a big cluster was obtained, which, despite being divided into six main clusters, they are very interconnected, indicating that all the keywords are related.
The first cluster is mainly associated with the thematic core of social issues, such as poverty, public health, education, gender equality, and reduced inequalities which reflects to the SDG 1, 3, 4, 5, and 10, respectively. The most frequent keywords in this cluster were policy, Africa, education, poverty, diseases and covid-19 with 28, 28, 16, 15, 12, and 12 occurrences, respectively. However, regarding their ANC, the most influential keywords was middle-income countries. Furthermore, despite this cluster having the largest number of keywords, its ANC is among the smallest. Thus, this is a strong indicative that social issues are not taken as seriously as those dealt with in other clusters.
The second cluster is divided in two main hubs, one for economic growth (SDG 8), production (SDG 12) and innovation (SDG 9) with the keywords Industry 4.0, big data, supply chain, technology, corporate social-reasonability, innovation, management, business, governance, circular economy; and one for climate action (SDG 13) and energy (SDG 7), with the keywords energy, impacts, climate. It is clear that energy and climate are well connected, as the growth of some industrial sectors directly affects the consumption and production of energy and consequently the impacts on the climate. Investigating the normalized words, a hidden key element was observed, the keyword industry 4.0 presented the highest ANC (3.99), which is 3.5 times higher than the average ANC of the analyzed keywords, followed by the keywords big data, supply chain, nexus, strategy, circular economy and technology.
Previous bibliometric studies have reported that research related to the SDGs is focused mainly on health (SDG 3), climate change (SDG 13), food (SDG 2), energy (SDG7), and sanitation (SDG 6) (Meschede 2020; Belmonte-Ureña et al. 2021; Yeh et al. 2022; Londoño-Pineda and Cano 2022; Sianes et al. 2022). Controversially, the present research found that academic research on SDGs 8, 9, 11, which was considered poorly researched and lacking in consolidated literature (Pizzi et al. 2020), obtained greater evidence, despite the number of occurrences and citations that can lead to an erroneous conclusion. This result is very important, as it is possible to observe the rapid growth and the importance of technology to achieve the goals, aiming at profitability and economic growth, in addition to pursuing environmental benefits (Walsh et al. 2020).
The third cluster is composed of a miscellaneous of SDG. However, regarding the occurrences of keywords, the keyword health (SDG 3) is central and is the most used keyword in this cluster with 37 occurrences, followed by water (SDG 6), cities (SDG 11), indicators, and sanitation (SDG 6). Furthermore, in regard to the ANC of the keywords, life-cycle assessment (SDG 12), waste management (SDG 11), tourism (SDG 8), industry (SDG 9) presented the higher scores. This result indicates that the most influential keywords in this cluster are related to economic growth, industry, innovation, and production, which are also linked to health conditions and the reduction of impacts on the planet, explaining the occurrence of the keywords, such as emission (SDG 13) and environmental impacts that are also part of this cluster (Rosa and Hassmiller 2020).
The keywords food security, agriculture, and food are the most used keywords in the fourth cluster and are associated with SDG 2. Bioenergy (SDG 7) and waste-water (SDG 6) are also present in the fourth cluster and can be also linked to the agricultural and/or food production. Some keywords could be attributed to SDG 15, such as land and soil. However, they possibly refer to land cultivation and therefore, are aiming at higher agricultural productivity, and thus, they were also classified as SDG 2. Furthermore, trade-offs and stakeholders were among the most evident keywords according to the ANC, and are related to the difficulties of implementing the SDG, being the only cluster where the 5Ps category partnership was found (SDG 17). This result indicate that applied research aiming at the implementation of the SDG is being developed.
The most prominent keyword in the fifth cluster is climate-change, with 53 occurrences, followed by ecosystem, adaptation, biodiversity, and gender. This cluster is well-defined and its issues can be assigned mainly to SDGs 13 and 15, and consequently to the 5Ps category, planet. Furthermore, regarding the ANC, it was noted that this cluster presented the lowest ANC, indicating the inferior relevance of the topic among the review articles studied.
No reviews articles related to marine water (SDG 14) were found (with a minimum number of 6 occurrences). Some authors (Bordignon 2021) also include freshwater in SDG 14, as being “life below water.” However, in the official document, the 2030 agenda only includes marine life (United Nations 2015). Many authors disagree with this distribution, which also causes some disagreements in the classification of SDG 6 on 5Ps, as people or planet. Tremblay et al. (2020) addressed this issue in his paper and proposed a relative distribution of the 5 Ps among the 17 SDGs according to the targets, whereas for SDG 6, about 40% corresponded to planet and 31% to people. Therefore, the lack of review articles on SDG 14 indicates a research gap in this field.
The sixth and last cluster includes the keywords, Electricity, greenhouse-gas, emissions, renewable energy, barriers, developing-countries that can be attributed to the SDGs 7, 10, and 13. International collaborative studies between developing countries and high-income countries were reported previously (Fig. 6). These collaborations are important for understanding the dynamics that affect developing countries due to the mixed and complicated impacts on achievement of SDG (Sianes et al. 2022). In this cluster, the most frequent keyword was developing-countries; however, the ones with the highest ANC were electricity and greenhouse-gas emission, while developing-countries was the one with the lowest ANC, indicating assuredly that the number of citations is not an adequate method for measuring subject relevance (Simko 2015).
In addition, the SDG 16, peace, justice, and strong institutions, showed to be a major gap in the research on SDG literature reviews. This topic is directly related to society, policies and governments, and plea for global peace. The targets also aim to reduce violence, corruption, bribe, exploitation, trafficking, torture, abuse, illicit arms, organized crime, and also develop international cooperation, participatory, and inclusive decision-making, inclusive (United Nations 2015).
Armed conflicts pose serious threats within the reach of the SDGs. As an example, the Russian-Ukrainian armed conflict is a dramatic world event that, in addition to the loss of life, impacts the environment, economy, and society. Both in countries directly involved, as well as in other countries, especially in developing countries, which are more vulnerable to the economic crisis. The negative regional and global impact could weaken the ability of many nations to achieve the SDGs by 2030, and could even make them unattainable (Pereira et al. 2022).
Conclusions
This bibliometric analysis provided the perspective of the status and evolution of research trend in the domain of SDG, with special emphasis on literature reviews regarding the fulfillment and consolidation of the agenda 2030. The revealed conclusions were:
An increasing trend in publications of SDG literature reviews was observed with a growing diversification in research areas.
Most of the review documents were categorized in general aspects of sustainability. In addition, most of relevant sources were associated with energy, multidisciplinary, and also general aspects of sustainability.
The most-cited documents and the most relevant sources indicated that the SDG with the greatest relevance was related to artificial intelligence (SDG 9) according to normalized citation analysis.
The results of bibliographic coupling demonstrated the lack of integration between the research, even when within the same research area.
England, the USA, and Australia were the most productive countries in SDG review articles. However, countries with intercontinental collaboration and with collaboration with developing countries showed greater relevance.
The terms with the highest values of keywords occurrences were health (SDG 3), climate change (SDG 13), food (SDG 2), energy (SDG 7), and water (SDG 6).
The keywords co-occurrence analysis spotted technology (SDG 9) and economic growth (SDG 8) as hidden key elements and with the greatest relevance among the analyzed keywords.
An imbalance between the different SDG was observed among academic research, and the main gaps found were for literature reviews on reduced inequalities (SDG 10), gender equality (SDG 5); oceans, seas, and marine environments (SDG 14); and related to peace, justice, and strong institutions (SDG 16).
Finally, the analysis revealed that the proposed methodology using bibliometric analysis of literature reviews presented different results when compared to previous reports in which a large number of documents was analyzed. The SDG is highly complex due to the many targets involved, trade-offs, and its multidisciplinary aspects. Therefore, future research should be directed to fulfill this gap through the development of more effective and reliable methods to verify the evolution of SDG in each area of research. Furthermore, the conceptualization and implementation of a theoretical framework that can be used in a generic way and also that describe standalone research areas to analyze the SDG considering their complexity will be also needed. In addition, it was observed that much research is still very disciplinary. Therefore, another future opportunity observed was the inclusion of more direct inter-linkages between certain SDG and practical applications. Thus, all research should be associated with the SDG, so that the efforts would be focused on achieving the agenda 2030, and a paradigm shift, practical, interdisciplinary actions with a system thinking perspective are necessary.
Author contribution
All authors contributed to the study conception and design. Investigation, methodology, data collection, and analysis were performed by Natália Ueda Yamaguchi, Eduarda Gameleira Bernardino, Maria Eliana Camargo Ferreira, Bruna Pietroski de Lima, Mauro Renato Pascotini, and Mirian Ueda Yamaguchi. The first draft of the manuscript was written by Natália Ueda Yamaguchi and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Funding
This work was supported by Cesumar Institute of Science, Technology and Innovation, Cesumar University (ICETI, Brazil).
Data availability
Raw data are available upon request.
Declarations
Ethical approval
Not applicable.
Consent to participate
Not applicable.
Consent to publish
Not applicable.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- Allen C, Metternicht G, Wiedmann T. Initial progress in implementing the Sustainable Development Goals (SDGs): a review of evidence from countries. Sustain Sci. 2018;13:1453–1467. doi: 10.1007/s11625-018-0572-3. [DOI] [Google Scholar]
- Bekker L, Alleyne G, Baral S, et al. Advancing global health and strengthening the HIV response in the era of the Sustainable Development Goals: the International AIDS Society-Lancet Commission. Lancet. 2018;392:312–358. doi: 10.1016/S0140-6736(18)31070-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Belmonte-Ureña LJ, Plaza-Úbeda JA, Vazquez-Brust D, Yakovleva N. Circular economy, degrowth and green growth as pathways for research on sustainable development goals: a global analysis and future agenda. Ecol Econ. 2021;185:107050. doi: 10.1016/j.ecolecon.2021.107050. [DOI] [Google Scholar]
- Bennich T, Weitz N, Carlsen H (2020) Deciphering the scientific literature on SDG interactions: a review and reading guide. Sci Total Environ 728:138405. 10.1016/j.scitotenv.2020.138405 [DOI] [PubMed]
- Bhattacharyya O, Wu D, Mossman K, et al. Criteria to assess potential reverse innovations: opportunities for shared learning between high- and low-income countries. Glob Health. 2017;13:4. doi: 10.1186/s12992-016-0225-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Biermann F, Kanie N, Kim R. Global governance by goal-setting: the novel approach of the UN Sustainable Development Goals. Curr Opin Environ Sustain. 2017;26–27:26–31. doi: 10.1016/j.cosust.2017.01.010. [DOI] [Google Scholar]
- Bordignon F. Dataset of search queries to map scientific publications to the UN sustainable development goals. Data Brief. 2021;34:106731. doi: 10.1016/j.dib.2021.106731. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bose S, Khan HZ. Sustainable development goals (SDGs) reporting and the role of country-level institutional factors: an international evidence. J Clean Prod. 2022;335:130290. doi: 10.1016/j.jclepro.2021.130290. [DOI] [Google Scholar]
- Buse K, Hawkes S (2015) Health in the sustainable development goals: ready for a paradigm shift? Glob Health 11:13. 10.1186/s12992-015-0098-8 [DOI] [PMC free article] [PubMed]
- Caiado R, Leal W, Quelhas O, et al. A literature-based review on potentials and constraints in the implementation of the sustainable development goals. J Clean Prod. 2018;198:1276–1288. doi: 10.1016/j.jclepro.2018.07.102. [DOI] [Google Scholar]
- Cobo MJ, López-Herrera AG, Herrera-Viedma E, Herrera F. Science mapping software tools: review, analysis, and cooperative study among tools. J Am Soc Inform Sci Technol. 2011;62(1382):1402. doi: 10.1002/asi.21525. [DOI] [Google Scholar]
- Díaz-López C, Martín-Blanco C, De la Torre Bayo JJ, et al. Analyzing the scientific evolution of the Sustainable Development Goals. Appl Sci. 2021;11:8286. doi: 10.3390/app11188286. [DOI] [Google Scholar]
- Ferreira FAF. Mapping the field of arts-based management: bibliographic coupling and co-citation analyses. J Bus Res. 2018;85:348–357. doi: 10.1016/j.jbusres.2017.03.026. [DOI] [Google Scholar]
- Fiorillo A, Maj M. The role of psychiatry in modern medicine. Int Rev Psychiatry. 2018;30:169–175. doi: 10.1080/09540261.2018.1436675. [DOI] [PubMed] [Google Scholar]
- Garrigos-Simon FJ, Narangajavana-Kaosiri Y, Lengua-Lengua I. Tourism and Sustainability: a bibliometric and visualization analysis. Sustainability. 2018;10:1976. doi: 10.3390/su10061976. [DOI] [Google Scholar]
- Hallinger P, Chatpinyakoop C. A bibliometric review of research on higher education for sustainable development, 1998–2018. Sustainability. 2019;11:2401. doi: 10.3390/su11082401. [DOI] [Google Scholar]
- Harris M, Dadwal V, Syed SB. Review of the reverse innovation series in globalization and health – where are we and what else is needed? Glob Health. 2020;16:26. doi: 10.1186/s12992-020-00555-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Khaled R, Ali H, Mohamed EKA. The Sustainable Development Goals and corporate sustainability performance: mapping, extent and determinants. J Clean Prod. 2021;311:127599. doi: 10.1016/j.jclepro.2021.127599. [DOI] [Google Scholar]
- Londoño-Pineda AA, Cano JA. Assessments under the United Nations Sustainable Development Goals: a bibliometric analysis. Environ Clim Technol. 2022;26:166–181. doi: 10.2478/rtuect-2022-0014. [DOI] [Google Scholar]
- Lund C, Brooke-Sumner C, Baingana F, et al. Social determinants of mental disorders and the Sustainable Development Goals: a systematic review of reviews. Lancet Psychiatry. 2018;5:357–369. doi: 10.1016/S2215-0366(18)30060-9. [DOI] [PubMed] [Google Scholar]
- Marzi G, Dabić M, Daim T, Garces E. Product and process innovation in manufacturing firms: a 30-year bibliometric analysis. Scientometrics. 2017;113:673–704. doi: 10.1007/s11192-017-2500-1. [DOI] [Google Scholar]
- McCollum D, Echeverri L, Busch S, et al (2018) Connecting the sustainable development goals by their energy inter-linkages. Environ Res Lett 13:033006. 10.1088/1748-9326/aaafe3
- Meschede C. The Sustainable Development Goals in Scientific literature: a bibliometric overview at the meta-level. Sustainability. 2020;12:4461. doi: 10.3390/su12114461. [DOI] [Google Scholar]
- Nerini F, Tomei J, To L, et al. Mapping synergies and trade-offs between energy and the Sustainable Development Goals. Nat Energy. 2018;3:10–15. doi: 10.1038/s41560-017-0036-5. [DOI] [Google Scholar]
- Nugent R, Bertram M, Jan S, et al. Investing in non-communicable disease prevention and management to advance the Sustainable Development Goals. Lancet. 2018;391:2029–2035. doi: 10.1016/S0140-6736(18)30667-6. [DOI] [PubMed] [Google Scholar]
- Pereira P, Zhao W, Symochko L, et al. The Russian-Ukrainian armed conflict will push back the sustainable development goals. Geogr Sustain. 2022;3:277–287. doi: 10.1016/j.geosus.2022.09.003. [DOI] [Google Scholar]
- Pizzi S, Caputo A, Corvino A, Venturelli A (2020) Management research and the UN sustainable development goals (SDGs): a bibliometric investigation and systematic review. J Clean Prod 276:124033. 10.1016/j.jclepro.2020.124033
- Prieto-Jiménez E, López-Catalán L, López-Catalán B, Domínguez-Fernández G. Sustainable Development Goals and education: a bibliometric mapping analysis. Sustainability. 2021;13:2126. doi: 10.3390/su13042126. [DOI] [Google Scholar]
- Randers J, Rockstrom J, Stoknes PE, Golüke U, Collste D, Cornell S (2018) Transformation is feasible: how to achieve the sustainable development goals within planetary boundaries. A report to the Club of Rome from Stockholm Resilience Centre and BI Norwegian Business School, Stockholm University. https://www.stockholmresilience.org/download/18.51d83659166367a9a16353/1539675518425/Report_Achieving%20the%20Sustainable%20Development%20Goals_WEB.pdf
- Romanelli JP, Gonçalves MCP, de Abreu Pestana LF, et al. Four challenges when conducting bibliometric reviews and how to deal with them. Environ Sci Pollut Res. 2021;28:60448–60458. doi: 10.1007/s11356-021-16420-x. [DOI] [PubMed] [Google Scholar]
- Rosa WE, Hassmiller SB. The Sustainable Development Goals and building a culture of health. AJN Am J Nurs. 2020;120:69–71. doi: 10.1097/01.NAJ.0000668772.33792.1f. [DOI] [PubMed] [Google Scholar]
- Santana M, Morales-Sánchez R, Pasamar S. Mapping the link between corporate social responsibility (CSR) and human resource management (HRM): how is this relationship measured? Sustainability. 2020;12:1678. doi: 10.3390/su12041678. [DOI] [Google Scholar]
- Sianes A, Vega-Muñoz A, Tirado-Valencia P, Ariza-Montes A. Impact of the sustainable development goals on the academic research agenda. A scientometric analysis. PLOS ONE. 2022;17:e0265409. doi: 10.1371/journal.pone.0265409. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Simko I. Analysis of bibliometric indicators to determine citation bias. Palgrave Commun. 2015;1:1–9. doi: 10.1057/palcomms.2015.11. [DOI] [Google Scholar]
- Stafford-Smith M, Griggs D, Gaffney O, et al. Integration: the key to implementing the Sustainable Development Goals. Sustain Sci. 2017;12:911–919. doi: 10.1007/s11625-016-0383-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tremblay D, Fortier F, Boucher J, et al. Sustainable development goal interactions: an analysis based on the five pillars of the 2030 agenda. Sustain Dev. 2020;28:1584–1596. doi: 10.1002/sd.2107. [DOI] [Google Scholar]
- United Nations (2015) A/RES/70/1 - Transforming our world: the 2030 agenda for sustainable development. Resolution adopted by the General Assembly on 25 September 2015, p 35. https://documents-dds-ny.un.org/doc/UNDOC/GEN/N15/291/89/PDF/N1529189.pdf?OpenElement
- United Nations (2017) A/RES/71/313 - Work of the Statistical Commission pertaining to the 2030 Agenda for Sustainable Development. Resolution adopted by the General Assembly on 6 July 2017, p 25. https://documents-dds-ny.un.org/doc/UNDOC/GEN/N17/207/63/PDF/N1720763.pdf?OpenElement
- van Eck NJ, Waltman L. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics. 2010;84:523–538. doi: 10.1007/s11192-009-0146-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Vanham D, Hoekstra A, Wada Y, et al. Physical water scarcity metrics for monitoring progress towards SDG target 6.4: an evaluation of indicator 6.4.2 “Level of water stress”. Sci Total Environ. 2018;613:218–232. doi: 10.1016/j.scitotenv.2017.09.056. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Vinuesa R, Azizpour H, Leite I, et al (2020) The role of artificial intelligence in achieving the Sustainable Development Goals. Nat Commun 11:233. 10.1038/s41467-019-14108-y [DOI] [PMC free article] [PubMed]
- Vrontis D, Christofi M. R&D internationalization and innovation: a systematic review, integrative framework and future research directions. J Bus Res. 2021;128:812–823. doi: 10.1016/j.jbusres.2019.03.031. [DOI] [Google Scholar]
- Walsh PP, Murphy E, Horan D. The role of science, technology and innovation in the UN 2030 agenda. Technol Forecast Soc Chang. 2020;154:119957. doi: 10.1016/j.techfore.2020.119957. [DOI] [Google Scholar]
- Wuni IY, Shen GQP, Osei-Kyei R. Scientometric review of global research trends on green buildings in construction journals from 1992 to 2018. Energy Build. 2019;190:69–85. doi: 10.1016/j.enbuild.2019.02.010. [DOI] [Google Scholar]
- Yakovleva N, Vazquez-Brust D. Stakeholder perspectives on CSR of mining MNCs in Argentina. J Bus Ethics. 2012;106:191–211. doi: 10.1007/s10551-011-0989-4. [DOI] [Google Scholar]
- Yeh S-C, Hsieh Y-L, Yu H-C, Tseng Y-H. The trends and content of research related to the Sustainable Development Goals: a systemic review. Appl Sci. 2022;12:6820. doi: 10.3390/app12136820. [DOI] [Google Scholar]
- Yu Y, Huang J (2021) Poverty reduction of Sustainable Development Goals in the 21st century: a bibliometric analysis. Front Commun 6:754181. 10.3389/fcomm.2021.754181
- Zhu D, Chen W, Qu X, et al. Future research needs for environmental science in China. Geogr Sustain. 2021;2:234–242. doi: 10.1016/j.geosus.2021.09.003. [DOI] [Google Scholar]
- Zupic I, Čater T. Bibliometric methods in management and organization. Organ Res Methods. 2015;18:429–472. doi: 10.1177/1094428114562629. [DOI] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Raw data are available upon request.