Skip to main content
PLOS One logoLink to PLOS One
. 2024 Nov 25;19(11):e0308183. doi: 10.1371/journal.pone.0308183

Organizational debt—Roadblock to agility in software engineering: Exploring an emerging concept and future research for software excellence

Osama Al-Baik 1,*, Mwaffaq Abu Alhija 2, Hikmat Abdeljaber 3, Muhammad Ovais Ahmad 4
Editor: Leander Luiz Klein5
PMCID: PMC11588203  PMID: 39585815

Abstract

In software engineering, organizational debt (OD) is a crucial but little-researched phenomena. OD refers to the accumulation of outdated structures, policies, and processes that hinder an organization’s advancement and adaptability. This multivocal literature review (MLR) synthesizes insights from software practitioners to elucidate OD causes, consequences, identification, and mitigation approaches that is considered a first step in illuminating the OD for software practitioners. After a thorough search, nine peer-reviewed articles and twenty-two recent blog posts on OD were included, indicating an emerging topic. Through inductive thematic analysis, four key topics emerged: definitions, causes like poorly managed change and siloed efforts, effects such as reduced innovation and agility, and mitigation strategies including agile principles, decentralized decision-making, and leveraging staff insights. While relying partly on non-peer-reviewed sources raises validity concerns, the review still provides a holistic and practical understanding of OD dynamics and complexities grounded in diverse perspectives. Further empirical research across diverse organizations would strengthen these preliminary findings. Effective OD management necessitates collaboration between academia and industry, considering technical debt (TD) best practices while tailoring interventions to OD’s distinct socio-technical characteristics.

1 Introduction

Software development is a dynamic field with many moving parts that result from the complicated interactions of many stakeholders, each of whom brings unique priorities and points of view to the table [1]. The complex character of software creation processes is further compounded by the unpredictable nature of human behavior in the development setting, as well as the particular problems posed by the different requirements of each project and the diverse backgrounds of all parties involved [2, 3]. Moreover, the relentless evolution of technologies and methodologies, interacting in unpredictable ways with organizational cultures and structures, introduces an additional layer of intricacy [4, 5].

The adoption of agile methodologies, emphasizing iterative development, constant collaboration with customers, and continual process improvements, represents an acknowledgement of the complexities inherent in contemporary software engineering [5, 6]. However, a critical concern emerges when development teams are treated as interchangeable resources subject to frequent reassignment between projects, disrupting team cohesion and knowledge transfer, potentially compromising productivity and quality [2]. Proponents of agile highlight the need for stable teams to realize the full benefits of iterative approaches and cross-functional collaboration [4].

While TD has garnered significant research attention, encompassing efforts, tools, types, management strategies, architectural aspects, agile development, and prioritization [79], there exists a noticeable paucity of exploration into the realm of OD. This paper aims to illuminate the phenomenon of OD through a MLR, synthesizing insights from software practitioners to uncover causes, consequences, identification, and mitigation approaches related to OD.

We conducted a search of Google Scholar in July 2023 with the keywords "organizational debt" AND "software". The search returned 120 results that were screened based on inclusion criteria of relevance to software engineering and organizational debt. A total of 22 sources met the criteria and were analyzed thematically to identify key topics related to causes, consequences, and mitigation strategies. As there were no academic articles on the OD and software, and in attempt to strengthen the research findings, another search was conducted in June 2024 with the keywords "organizational debt" AND (management OR "organizational behavior" OR "information systems"). This search resulted in including nine articles.

The subsequent sections of this pare are organized as follows, section 2 presents the background, section 3 explains the research methodology, section 4 reports the results, while section 5 discusses the validity assessment, section 6 dives into an in-depth discussion, and finally section 7 provides conclusions gleaned from the rigorous literature analysis.

2 Background

The software development process is a multifaceted and intricate undertaking characterized by the complex interplay of numerous stakeholders, each contributing to the dynamism of the environment. The unpredictability of human behavior within this context, coupled with the unique challenges presented by diverse project requirements and the distinct backgrounds of the individuals involved, further amplifies the complexity of the software development landscape [2]. The continuous advancement of technology and organizational procedures adds complexity, as these factors interact in unforeseen and frequently mutually beneficial ways.

Implementation of dynamic Agile and lean methodologies, characterized by iterative value delivery to clients and the elimination of wasteful practices that impact productivity and quality, acknowledge this complex nature [4, 6, 10, 11]. However, a significant issue occurs when individuals are perceived as replaceable "resources" who can be often reassigned to different positions. The empirical research clearly demonstrates that this technique is completely unproductive and has a detrimental impact on team relations for both the sender and the recipient. This poses a challenge to the transmission of information and, at times, even puts overall productivity and quality at risk.

Another challenge is enforcing a rigidly inflexible process, which sometimes gives managers certain bureaucratic complacencies between them, erroneously clinging to the belief that this very same process by itself could be some cure for all possible worse-case scenarios ever. Inevitably, such rigidity leads to blaming the team for deviations from a rigidly prescribed process when unexpected challenges ultimately arise a cultural blockade that frustrates continuous improvement [2].

The prevalent “Just get it done!” attitude though arising out of the desire for rapid development, works as a main source in software heterodony. As Cunningham pointed out in 1992 [12], there is a degree of borrowing enclosing code for the first time that resembles taking out a loan, and while little debt may fasten development, this must be promptly paid back with refactoring. There is no equivalent nefariousness of the TD, as each and every minute spent with a code that otherwise contains might be aggregated into interest on this type of financial loan. This definition was furthered by Avgeriou et al. [7] to include a range of TD artifacts, consisting of architectural debt and code as well as test-type debt.

TD has been studied in extreme depth from various perspectives, including efforts, tools, types, management strategies, architectural aspects, and agile development or prioritization [8, 9, 1318]. For example, 78 causes and 66 effects of TD have been identified by Rios et al. [15], considering a series of reasons, from tight timeframes to poor documentation or emphasis on high volumes over quality. Avgeriou et al. [16] made comparisons of tools for quantifying TD, noting the influence on maintainability, quality, project expenses, financial losses, code refinement, team overload, and dissatisfaction of the stakeholders. In particular, the study highlights the importance of management principles that can help manage architectural debt effectively, especially in a scenario where agile software development is involved.

In addition to TD, the notion of Non-Technical Debt (NTD) has been introduced to encompass various organizational debts that are not technical in nature [14, 19]. These debts include process, social, human, and cultural in nature. NTD arises from imprudent decision-making that prioritizes immediate benefits over long-term sustainability, thereby impeding numerous software development endeavors and influencing the outcome of projects. In consideration of the literature, the principal NTD types are examined and outlined below.

Process debt, a type of suboptimal conduct that provides immediate advantages but has enduring repercussions, may result from conducting these meetings solely for the purpose of updating statuses, thereby restricting their complete potential. This is largely attributable to a lack of process competencies, a gap resulting from a subtle deviation from the norm, and external influences such as technological tools and trends [19]. People debt refers to development matters such as owed specialization on a limited number of staff accruing from late training and recruitment. Those factors leading to people debt are suggested a lack of knowledge, experience, devotion, and psychological safety; inappropriate management decisions acceptance issues among personnel or diminutive developer reasonable confidence [19]. Cultural debt. Technical decisions that lead to delinquency to culture may consume a significant portion or the entire culture of an organization. This can potentially result in the division of teams, deterioration of communication practices, and reduced effectiveness of leadership [14]. Social debt may be detrimental to the suboptimal development environment of numerous software development communities, as evidenced by gender bias resulting from communication and collaboration constraints [19].

Notably, with the advent of novel hybrid work policies in contemporary organizational environments, managers may encounter the peril of OD. Policies of this nature, which are instrumental in shaping the development of emerging norms among employees, necessitate meticulous scrutiny to avert unfavorable consequences. Liu et al. [20] identified and analyzed the impact of twenty-three mechanisms, which they categorized into eight coordination domains and implemented in shared mental models, team coordination, cohesion, and learning. The software development process is inherently complicated and requires a multidimensional approach that considers the specific characteristics of each project’s participants and their changing technological positions during organizational process transitions. By highlighting the importance of delivering value, waste reduction, and adaptability, the combination of Agile and Lean approaches can provide a solution to these difficulties [4, 6, 10, 11].

3 Research method

One of the tools used to handle various issues that arise in the field of software engineering is the MLR, This study employed MLR approach, which combines the analysis of practitioner literature (grey literature) with academic sources to provide a comprehensive understanding of the investigated phenomenon [21]. This method includes a broad spectrum of written sources that are easily accessible, considers several authors’ perspectives, and even touches upon non-scientific subjects. For MLR to be applied effectively, it is important for a specific research question to be formulated since MLR involves manifold points of view. Our study aimed to achieve a deep understanding of how software practitioners articulate the concept of ‘organizational debt’. In an attempt to ensure rigorous research, the authors used the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA), to report the research items (see [S1 Checklist]).

3.1 Initial grey literature search

The deliberate source identification approach was used in this MLR as Garousi and Mantyla suggested [21]. The Google search engine was used as a tool of initial search instead of traditional scholarly databases like SpringerLink, ACM, and IEEE eXplore. This conviction was based on the fact that the selected topic for investigation, OD, is quite new and there are few sources of the relevant literature about this issue. The search capabilities of Google made it possible to efficiently search for the grey literatures. In July 2023, we did a literature search using a meticulously designed query ("organisational debt" OR "organizational debt" OR "OD" AND "software") to guarantee a thorough scope.

The inclusion of the term "software" in the search string was deliberate, encompassing studies related to software, software development, software engineering, or software-intensive products, services, and systems. The dual inclusion of ("OD" OR "organizational debt") aimed to encompass all pertinent literature on OD.

A total of 120 sources from the initial search results were scrutinized, aligning with common practices in grey literature research, where researchers often consider the first 50 search results [22]. Each link was meticulously examined, and relevant information was documented. Exclusions were made for sources not in English, videos, advertisements, catalogues, duplicates, research profiles, or those outside the realm of software engineering. Ultimately, twenty-two blog posts from the selected 120 sources were deemed pertinent and are duly cited in the reference list from [2344], inclusive. Notably, the search strategy did not yield any academic relevant articles. Out of curiosity, we conducted a manual search of Scopus, Web of Science, PubMed, SpringerLink, ACM, and IEEE eXplore. Only one result was obtained, and it was co-authored by one of the co-authors of this study.

3.2 Academic literature search

To address the limitation of relying solely on practitioner perspectives and enhance the theoretical grounding of the findings, a comprehensive search for academic literature was conducted. The following databases were searched: Scopus, Web of Science, ABI/INFORM, JSTOR, and Google Scholar.

We conduct a thorough search for relevant academic literature on OD from related disciplines, such as management, organizational behavior, and information systems. This will broaden our data corpus and provide a more comprehensive view of the phenomenon. The search strings used were:

("organizational debt" OR "organisational debt") AND (management OR "organizational behavior" OR "information systems")

The searches were limited to peer-reviewed journal articles and conference proceedings published in English between 2000 and 2023. The initial search yielded 387 results. The following inclusion criteria were applied to the academic literature search results:

  1. Empirical studies or conceptual/theoretical papers explicitly addressing organizational debt or closely related concepts (e.g., management debt, organizational inertia, organizational agility).

  2. Studies focusing on the causes, consequences, or mitigation strategies related to organizational debt or similar phenomena.

  3. Studies from the domains of management, organizational behavior, information systems, or related disciplines.

The exclusion criteria were:

  1. Papers not published in English.

  2. Papers not available in full text.

  3. Papers focusing solely on technical debt without addressing organizational or non-technical debt aspects.

After applying the selection criteria, a total of 9 academic papers were included in the final analysis and are duly cited in the reference list from [4553], in addition to the 22 blog posts [2344] from the initial grey literature search [S1 Checklist].

3.3 Data analysis

The subsequent data analysis phase employed thematic analysis techniques, focusing on four emergent themes: OD definitions, OD causes, consequences, and mitigation strategies. Unlike a deductive approach with pre-defined themes or codes, our analysis embraced an inductive stance, allowing themes related to causes, effects, and mitigation strategies to surface organically from the data. The utilization of this open analysis methodology amplifies the number and profundity of ideas derived from the literature.

Findings from both the practitioner literature and academic sources were synthesized and integrated within each theme, facilitating a comprehensive understanding of the phenomenon from diverse perspectives. By combining the analysis of grey literature with academic sources, this MLR provides a holistic and multi-faceted exploration of organizational debt, grounded in both practical insights and theoretical foundations. The findings of our MLR analysis are presented in Section 4.

4 Results

In this section, we conduct a comprehensive examination of the results obtained from the MLR. The primary objective is to examine the academic literature and progression of grey literature publications and gather diverse explanations of OD. Furthermore, we will investigate the fundamental causes and outcomes of OD, and subsequently evaluate strategies for detecting and reducing it.

4.1 Organizational debt concept in related fields

Based on the analysis of the identified academic literature, we have synthesized the key findings related to organizational debt (OD) causes, consequences, and mitigation strategies.

4.1.1 Causes of organizational debt

  1. Outdated organizational structures and policies: Failure to adapt structures, processes, and policies to changing business environments and market conditions can lead to the accumulation of OD [45, 48, 49].

  2. Short-term decision-making: Prioritizing short-term gains over long-term sustainability and strategic objectives can result in compromises that contribute to OD [47, 48].

  3. Lack of organizational agility: Rigid hierarchies, siloed operations, and resistance to change hinder an organization’s ability to respond effectively to emerging challenges and opportunities, leading to OD accumulation [46, 51].

  4. Inefficient coordination and collaboration: Inadequate mechanisms for cross-functional coordination and knowledge sharing can impede productivity and innovation, contributing to OD [46, 52].

  5. Technology-related stressors: Overreliance on technology, inadequate technology governance, and technostress can negatively impact employee productivity and well-being, leading to OD [50, 53].

4.1.2 Consequences of organizational debt

  1. Reduced productivity and efficiency: OD can result in suboptimal processes, redundancies, and inefficiencies, negatively impacting overall organizational productivity [45, 48, 52].

  2. Diminished innovation and competitiveness: Outdated structures, policies, and processes can stifle creativity, hinder adaptability, and undermine an organization’s ability to innovate and remain competitive [47, 49].

  3. Employee disengagement and turnover: A dysfunctional work environment, lack of empowerment, and technostress can lead to decreased employee engagement, job satisfaction, and increased turnover [54, 55].

  4. Financial implications: OD can result in increased operating costs, missed opportunities, and potential financial losses due to inefficiencies and lack of competitiveness [45, 51]

  5. Reputational damage: Failure to address OD and its consequences can harm an organization’s reputation, customer satisfaction, and stakeholder trust [49].

4.1.3 Mitigation strategies for organizational debt

  1. Continuous organizational assessment and adaptation: Regular audits, employee feedback, and performance monitoring can help identify sources of OD, enabling proactive adjustments to structures, processes, and policies [48, 49].

  2. Fostering organizational agility and flexibility: Implementing agile methodologies, decentralized decision-making, and cross-functional collaboration can enhance an organization’s ability to respond to change and mitigate OD accumulation [46, 51].

  3. Strategic alignment and long-term planning: Ensuring alignment between organizational structures, processes, and strategic objectives, while considering long-term implications of decisions, can prevent OD accumulation [47, 48]

  4. Technology governance and digital transformation: Implementing effective technology governance, digital upskilling initiatives, and addressing technostress can mitigate technology-related OD drivers [50, 53]

  5. Empowering employees and promoting a learning culture: Encouraging employee autonomy, continuous learning, and innovation can foster an engaged workforce and support OD mitigation efforts [49, 50].

This summary highlights the multifaceted nature of OD, encompassing structural, cultural, technological, and human factors. Effective OD management requires a holistic approach that addresses the underlying causes, mitigates the negative consequences, and promotes organizational adaptability, innovation, and employee well-being.

4.2 Trend and definition of organizational debt concept

The MLR systematically assembled twenty-two blog postings centered on OD, authored by seasoned professionals in the software engineering field. A clear pattern emerges as six out of 13 articles were written in the past couple of years 2020–2023, highlighting the increasing discussion surrounding the OD phenomenon. The conceptual roots of OD trace back to 2015 when Steve Blank extended the metaphor of TD and characterized OD as "worse" [35].

Notably, this idea finds antecedents in Ben Horowitz’s conceptualization of "management debt" dating back to 2012 [34]. Subsequent contributors like Dignan broadened the scope, asserting that OD is not confined to start-ups but holds significance on a broader organizational scale [32]. The MLR results offer a plethora of OD definitions, reflecting the nuanced perspectives of software industry professionals, Table 1 below shows a summary of these definitions:

Table 1. OD definitions.

Proposed Definition Year Source
"OD is all the people/culture compromises made to ’just get it done’ in the early stages of a start-up" 2015 [35]
"Organizational debt is any structure or policy that no longer serves an organization" 2020 [30]
"Organizational debt is the accumulation of changes and decisions leaders should have made but did not" 2016 [33]
“The interest companies pay when their structure & policies stay fixed and/or accumulate as the world changes" 2016 [32]
"Management Debt is incurred when you make an expedient, short-term management decision with an expensive, long-term consequence" 2022 [24]
"Organizations may intentionally or unintentionally incur organizational debt through management actions, governance process changes, internal process changes, or large-scale organizational changes when short-term advantages are sought at the expense of ’doing things right” 2017 [36]
"Organizational debt is the baggage that prevents people from delivering astonishing results" 2015 [28]
"Organizational debt—our organizations are also a good excuse to avoid changes, as we often look for someone who is going to help us, but we do not really want to give him or her the power to implement the changes" 2019 [29]
"Organizational debt is sibling of technical debt, for example a toxic culture, struggling leader etc." 2020 [27]
"Organizational debt: things that should’ve been done to ensure health & efficiency, but weren’t" 2021 [41]
"Organizational debt, an analogy! During the execution of organizational changes (transformations, reorganizations, changes in ways of working etc.) shortcuts are taken that lead to frustration, more time and money etc. It’s the same thing as technical debt" 2021 [23]
"Organizational Debt is the interest companies pay when their structure and policies 1) stay fixed and/or 2) accumulate as the world changes" 2016 [39]
"Organizational debt is a holistic concept, and it is more than technical debt and also different from bureaucracy. Organizational debt is a networked concept that fosters the blame-free identification of cross-functional and cross-department weak points" 2023 [43]

The literature review revealed 13 distinct definitions of organisational debt proposed by various practitioners and experts. While differing in precise wording and focus, several core concepts recur across these definitions. Organisational debt refers to: 1) Accumulation of unresolved decisions, unimplemented actions, and compromises made by leaders [3335]. 2) Outdated organisational structures, policies and processes that no longer serve the organisation’s objectives [23, 30]. 3) The gap between the organisation’s strategic plans and actual capacity to implement them [25, 26]; and 4) Interest paid in the form of reduced efficiency, innovation, and adaptability [32, 39]. Synthesizing these recurring concepts, this study defines organisational debt as:

  • " Organizational debt is the accumulation of outdated structures, policies, and processes that are no longer advantageous for the organization. Consequently, obstructs the advancement of the organization and its capacity to adjust to evolving conditions, ultimately hindering its potential for excellence.

This concept encompasses the fundamental aspects of OD that have arisen from the literature. Firstly, it appears in outdated organizational elements that do not correspond with strategic goals. Furthermore, it hinders the advancement of the organization and its ability to adjust to new circumstances. This succinct yet all-encompassing description establishes a conceptual basis for discussing the origins, effects, and administration of OD.

4.3 Causes and consequences of organizational debt

The MLR identified multiple primary factors contributing to the accrual of OD. These causes indicate inadequacies in leadership, culture, teamwork, and change management. Table 2 summarizes the key causes to and consequences of OD:

Table 2. OD causes and consequences.

Cause of OD Consequences of OD
Poorly managed organizational change Reduced efficiency, productivity, innovation and morale
Lack of collaboration culture Lack of agility to adapt to market changes
Siloed change efforts Reduced competitiveness and stagnating growth
Avoiding confrontation of issues Entrenched bureaucracy and resistance to change

Poorly Managed Organizational Change: Undertaking significant modifications without sufficient preparation, instruction, and involvement of relevant parties frequently results in the failure to attain the intended results. Insufficient communication of the rationale and benefits of introducing new tools or processes can lead to confusion among staff. Inadequate training on new systems leaves employees struggling to adapt, reducing productivity and morale. Failure to involve key stakeholders early in the change initiative overlooks valuable insights and acceptance. Poor change management spreads dissatisfaction and resistance, distracting focus from regular operations. Ultimately, the change initiative falls short of goals and disrupts workflows. This poorly managed change incurs OD through reduced efficiency, disengaged staff who resent imposed changes, and lack of flexibility to keep pace with business needs [31, 37].

Lack of Collaboration Culture: Excluding staff from decision-making and change initiatives overlooks valuable perspectives and insights from those closest to the work. A lack of openness to bottom-up input and ideas hinders innovation. Silos between teams or departments prevent knowledge sharing which can highlight process inefficiencies and redundancies. Leaders who resist input from below risk disenfranchising employees, reducing engagement and motivation. Poor collaboration limits the organization’s ability to leverage its collective intelligence fully. Instead of synergistic workflow, fragmented efforts reduce process efficiency, agility, and continuous improvement [36].

Siloed Change Efforts: When different departments or teams undertake changes in isolation, this leads to duplicated efforts, incompatible systems, and wasted resources. A lack of communication and coordination between disparate change initiatives produces fragmented outcomes. Divergent tools, policies, and processes increase complexity across the organization. This results in decreased productivity, since efforts to resolve misalignments and conflicts between departments are diverted from productive endeavors. Additionally, redundant endeavors signify neglected prospects to maximize the utilization of communal resources and expertise. Without an integrated approach to change management, inferior outcomes are inevitable [31, 36].

Avoiding Discontent: A reluctance to confront subpar performance and opposition to essential modifications contribute to the continuation of inefficiency. This acceptance of mediocrity becomes the norm when supervisors are unable to approach troublesome employees for fear of retaliation. When high achievers passively labor with underachievers, they are burdened. By evading necessary yet disruptive adjustments for fear of inciting complaints, one can perpetuate antiquated methods and misalignments. Although the immediate benefit may be to preserve peace, the long-term repercussions of continued mediocrity and stagnation are considerably more severe. An organization incurs debt as challenging matters continue to accrue unsolved [33, 34]. These root causes interact to produce a range of detrimental consequences for software organizations:

Reduced efficiency, productivity, innovation and morale: diminished efficacy, output, ingenuity, and team spirit [28, 32]. Displeased employees perceive their contributions as underappreciated and their time squandered on repetitive tasks or navigating complex administrative procedures.

Lack of agility and inability to adapt to market changes: insufficient flexibility and incapacity to adjust to shifts in the market [25, 26]. The organization’s outdated legacy systems and bloated processes hinder its ability to stay up with more agile competitors, making it slow and inflexible.

Reduced competitiveness and stagnant growth [25, 29]. The presence of TD and inefficient procedures necessitates a significant allocation of resources, which in turn hampers the ability to invest substantially in innovation.

The presence of a deeply rooted administrative system and opposition to altering it [25, 34]. The accumulation of organizational complexity persists as people adhere to familiar heritage systems. Efforts to initiate change struggle as employees have a sense of powerlessness.

Therefore, OD incurs major “interest” costs from reduced productivity, innovation lag, technology debt, opportunity costs from forfeited growth, and an increasingly dysfunctional workplace. Software leaders must recognize this self-reinforcing downward spiral and intervene promptly to restore organizational fitness and strategic alignment.

4.4 Identification and mitigation of organisational debt

Identifying and mitigating organisational debt requires a systematic approach given its multifaceted nature. Organisational debt symptoms can be spotted through regular performance monitoring, employee surveys, and audits of processes [38, 41]. Comprehensive evaluation of various organisational components is vital for deeper insights [40].

Quantitative performance metrics offer warning signs such as prolonged declines in productivity, increasing software defects, lags in new feature release, product quality issues, and rising customer complaints. Comparing metrics over time and against competitors highlights underperformance. Periodic audits help assess process efficiency, redundancy, and alignment with objectives. Surveys and interviews to gather employee perspectives on pain points complement the top-down analysis. The utilization of both quantitative and qualitative data allows for the cross-validation of findings regarding the state of organizational components.

Signs of insufficient regulations, protocols, and organizational norms encompass role ambiguity, decision-making barriers, unbalanced incentives, restricted openness, and isolated information. Process debt can be identified through the confluence of static production metrics and employee discontentment stemming from an overabundance of bureaucratic procedures. The TD resulting from the deterioration of product quality, as evidenced by developers’ grievances regarding impracticable deadlines, underscores the need for a reassessment of administrative objectives. Through this methodical evaluation of symptoms, problem areas that necessitate improvement are identified.

By placing value delivery above hierarchical structures and control processes, software teams may concentrate on rapidly satisfying stakeholder requirements with shippable increments. The adoption of agile concepts, including the establishment of autonomous, cross-functional, and small teams, serves to mitigate the occurrence of isolated tasks and empowers personnel to devise strategies and execute them autonomously. Minimally feasible procedures provide an adequate degree of structure to facilitate outcomes, while avoiding the development of too intricate bureaucratic systems. Complete visibility of the status of work is achieved through the utilization of information radiators, boards, and charts, hence augmenting transparency. This promotes confidence, shared accountability, and reciprocal liability for results. Efficient procedures that prioritize initial advantages foster innovation, adaptability, and the capacity to maintain competitiveness in the face of adversity [2, 3].

Stagnation can be reduced by fostering role creation and using flexible work practices, which enable people to customize their responsibilities in accordance with their skill sets. Micromanagement has a detrimental effect on employee motivation since it communicates a deficiency in confidence and recognition of their abilities, which in turn stifles the innovative thinking required to resolve complex problems. Providing teams with the autonomy to devise their own strategies for attaining mutually agreed-upon objectives enhances their degree of engagement. When individuals avoid from erecting hard demarcations between positions, they are better able to readily adapt to changing circumstances and respond with adaptability to emerging issues. It is possible to cultivate a culture of innovation inside an organization that fosters psychological safety by encouraging individuals to take risks and gaining important insights from failures. It is possible to encourage the innate motivation of individuals and foster flexibility by offering deliberate autonomy to individuals rather than enforcing strict conformance to customary processes [2].

Continuous feedback from frontline personnel provides timely insights into inconsistencies that may occur between stated protocols and practical scenarios. These inconsistencies may appear in a variety of situations. Traditional surveys that are anonymous are able to obtain honest responses without the risk of negative consequences. Town hall meetings create an atmosphere that encourages the free and open airing of feelings of dissatisfaction and the sharing of potential solutions. Those in leadership positions who aggressively seek criticism demonstrate a willingness to receive input and a dedication to working together to find solutions to problems. This encourages symbiotic relationships with both employees and management, as opposed to transactional interactions that are more likely to be characterized by mistrust. Opportunities to modify systems in a more effective manner that promotes human flourishing can be discovered through a critical evaluation of disparities between objectives and results [2].

By involving employees in decentralized and participatory decision-making, policies and processes may be flexibly adapted to evolving realities. Absent extensive stakeholder input, systems intended for abstraction frequently deviate from requirements. Nobody comprehends uncertainties and sources of friction more thoroughly than personnel who are immersed on a daily basis. The insights they possess are often overlooked by questionnaires due to their personal experiences. Architectures based on collaborative choice and co-creation foster psychological investment and alignment. Individuals performing the tasks are in the best position to engage in open problem-solving as obstacles arise via cycles of experimentation, feedback, and learning [2, 3].

Monitoring metrics on performance, behavior, and innovation provides a holistic dashboard. Productivity goals have potential to dehumanize work into joyless drudgery if not complemented by gauges of employee experience, health, and ability to creatively contribute. Well-rounded assessments consider both business and human needs. For instance, software teams require sufficient time for refactoring TD to maintain velocity long-term. Quality cannot be sacrificed for feature output volume. Regular pulse checks on morale, trust, purpose, and work-life balance offer leading indicators before burnout [11].

Implementing programs that allow frontline employees to anonymously identify mismatched policies helps prevent the divergence of systems from actual demands. Providing incentives for constructive critiques encourages the identification of defective processes at an early stage, thus preventing the accumulation of debt. Encouraging disagreement exhibits a sense of psychological security to question the existing state of affairs. Leaders must acknowledge that policies are not permanent and need to be regularly reassessed as circumstances change. The key differentiating factor between learning organizations and stagnant bureaucracies is the ongoing evaluation of whether current structures and norms effectively meet the evolving needs. This process of continuous reflection is emphasized in learning organizations [2, 56].

In summary, continuously tuning processes and structures requires decentralizing power, trusting employees, and weighing quantitative metrics against qualitative insights. An agile mindset focused on early delivery of value, adaptation in response to learning, and leveraging collective intelligence helps curb debt. Leaders play a key role in diagnosing and addressing sources of debt through organizational redesign. However, they often lack deep visibility into the causes of inertia, misalignment, and inefficiency on the frontlines. Engaging staff in open discussions to uncover pain points is invaluable, combined with a culture of psychological safety where people feel secure speaking up. Dismantling OD requires placing human needs on par with business needs.

4.5 Future research directions

While this literature review has consolidated understanding of OD, several fruitful avenues exist for further investigation based on current knowledge gaps:

  • Develop metrics to quantify OD, enabling rigorous tracking and benchmarking. Combine productivity data with indicators of culture, innovation, and TD [52]. Establish validated scales to measure dimensions like employee engagement, psychological safety, organizational agility, and leadership effectiveness [5759]. Statistical modeling can relate these metrics to OD.

  • Conduct empirical studies on the impact of OD on workforce motivation, attrition, fatigue, and burnout [6062]. Use questionnaires and ethnographic methods to gather insights. Relate debt to tangible individual performance metrics like productivity, absenteeism, and error rates.

  • Investigate through case studies the relationship between OD and customer satisfaction [63, 64], especially in software-intensive service organizations. Survey data can correlate debt to metrics like call resolution times, complaint rates, churn, and net promoter scores.

  • Examine through controlled experiments the role of OD in software project success/failure [6567]. Vary team structures and processes to reveal optimal configurations. Productivity, quality, cost, and schedule metrics assess performance.

  • Estimate the economic costs of OD through case studies and cost-modeling across software companies [6870]. Assess opportunity costs from delayed innovations. Relate to total cost of ownership models.

  • Explore whether TD quantification techniques [7173] can be extended to provide estimates of OD. Compare their accuracy and utility.

  • Design field studies of interventions such as restructured teams, revised workflows, and new planning processes to validate OD mitigation techniques [7476]. Measure before and after effects.

Further research to address these gaps will provide more rigorous, empirically-grounded insights to guide debt management in practice. It represents an emerging interdisciplinary arena spanning management science, organizational behavior, anthropology, and software engineering [77]. Collaboration between academics and industry practitioners is needed to develop context-specific strategies rooted in both theory and pragmatism [78]. There are rich possibilities for cross-pollination between disciplines to uncover novel solutions [79]. With organizational agility and adaptability growing ever more crucial in turbulent conditions [80], understanding how to minimize friction and debt represents the key to sustaining innovation and competitiveness [56].

5 Validity threats and methodological limitations

This section examines the constraints of the study design and any factors that could undermine the accuracy and reliability of its results. This MLR follows the methodological framework proposed by Garousi and Mantyla [21]. It conducts a thorough assessment of the external, internal, and construct validity, while also recognizing the inherent limits of the research process.

External Validity Threat: One of the significant risks is associated with the lack of the peer-review process in the practitioner literature. Grey literature offers practical information that is acquired from an industrial setting. Still, such absence of a broad empirical basis warns of a potential threat of built-in bias. The contextual details of the described experiences are not clear. As such, the findings cannot be generalized to software enterprises at large. This study strengthening the results with more analysis from studies published in peer-reviewed journals to support the conclusions. In addition, the limitation to English primary sources raises the concern about the degree of accessibility within other areas of language. Despite this drawback, it is reasonable to state that MLR covers a large body of information, thus facilitating the representation of the views of both TD and NTD experts in a fully adequate manner.

Internal validity Threat pertains to the evaluation of potential biases that may influence the outcomes of the MLR analysis. The primary objective is to identify and mitigate these biases. Employing a pre-established search engine, specific search phrases, and well-defined inclusion/exclusion criteria as components of a systematic source selection process is a dependable strategy to ensure consistent results. However, following careful analysis, one can detect inherent dangers linked to limitations on search terms, reliance on a solitary search engine, and potential biases in the execution of inclusion/exclusion criteria. To tackle these problems, a structured search was carried out using precise keywords in a methodical fashion, aiming to reduce the chance of overlooking pertinent studies. The research team, comprising two seasoned software engineering researchers, and a third researcher to be an auditor bolsters internal validity by using their extensive expertise and educational qualifications, thereby mitigating potential risks to some extent.

Construct Validity Threat: One of the major challenges to construct validity is the absence of empirical evidence in primary research. The dependence on gray literature, which mostly consists of subjective opinions and practical knowledge from professionals, poses a possible risk. Although this type of information provides a significant insight into real-world industrial situations, the lack of specific contextual elements raises an epistemological question about the clarity of the knowledge source [21]. However, it is important to acknowledge that this constraint, arising from the characteristics of grey literature, does not automatically render the findings erroneous. A proposed solution is envisioning a situation where the identical professionals, who have expressed their viewpoints in unofficial publications, do structured interviews or surveys on the identical topic. Adopting such a strategy would probably produce reliable outcomes, however with an increased level of methodological precision.

The use of blog posts and online articles inherits further validity threats. The expertise and qualifications of authors are difficult to ascertain. Sample size was small, constrained to only English search results. Relevant publications in other languages were excluded. The anecdotal nature of this literature reduces reliability compared to large-sample studies. However, the findings still offer useful preliminary insights on an emerging topic with minimal academic attention currently. Despite limitations, this exploratory MLR serves the intended purpose of mapping key concepts, issues, and questions on organisational debt to guide future research.

To summarize, the MLR effectively deals with validity threats and limitations, providing valuable insights into the complex field of software engineering challenges. However, it is crucial to continuously consider the methodology and prioritize transparency in order to improve research efforts in this area.

6 Discussion

This study identifies OD as a complex problem, closely connected to factors such as corporate culture, leadership dynamics, collaborative frameworks, process design complexities, decision-making paradigms, and organizational structures. This study synthesized practitioner and academic perspectives to clarify the complex phenomenon of organizational debt (OD) within the software engineering context. By integrating insights from grey literature and peer-reviewed sources, our analysis revealed both convergent and divergent viewpoints, enriching our understanding of OD dynamics.

6.1 Convergence of perspectives

Both practitioner and academic sources acknowledged the adverse consequences of OD, including reduced productivity, innovation, and competitiveness [28, 32, 45, 48, 49]. The accumulation of outdated structures, policies, and processes was consistently identified as a root cause, reflecting the organization’s inability to adapt to changing environments [23, 30, 33, 45, 48, 49]. Furthermore, there was a consensus on the detrimental impact of siloed operations, lack of cross-functional collaboration, and resistance to change, which fuel the growth of OD [30, 35, 51, 79].

6.2 Divergence of perspectives

While practitioner sources emphasized the role of leadership decisions, compromises, and avoidance of disruptive changes in contributing to OD [3335], academic literature investigated the organizational and human factors. Aspects such as rigid hierarchies, ineffective coordination mechanisms, technostress, and employee disengagement were highlighted as significant OD drivers [46, 50, 52, 53]. This divergence underscores the multifaceted nature of OD, which extends beyond individual decision-making to encompass systemic organizational and technological complexities.

6.3 Implications for software engineering research

The integration of practitioner and academic viewpoints opens up avenues for future research in software engineering. Empirical investigations into the interplay between OD, software project outcomes, and organizational performance are warranted. Quantitative studies could explore the economic impact of OD, complementing existing technical debt quantification techniques [7173]. Qualitative inquiries could unravel the socio-technical dynamics underpinning OD accumulation and its relationship with organizational culture, leadership, and team dynamics.

Furthermore, interdisciplinary collaborations with management science, organizational behavior, and information systems researchers hold promise for cross-pollinating insights and developing holistic OD management frameworks tailored to software-intensive organizations. Longitudinal studies tracking the effectiveness of interventions, such as restructuring, process redesign, and cultural transformation, could validate proposed mitigation strategies empirically.

6.4 Implications for software engineering practice

This study’s findings underscore the importance of proactive OD management for software organizations. Practitioners should prioritize continuous assessment and adaptation, fostering organizational agility and flexibility [48, 49, 51]. Regular audits, employee feedback, and performance monitoring can aid in identifying OD hotspots and informing timely interventions.

Aligning organizational structures, processes, and strategic objectives is crucial, while considering the long-term implications of decisions [47, 48]. Effective technology governance, digital upskilling initiatives, and addressing technostress can mitigate technology-related OD drivers [50, 53].

Moreover, empowering employees, promoting a learning culture, and encouraging cross-functional collaboration can strengthen an organization’s collective intelligence and capacity for innovation [49, 50]. Leaders should embrace decentralized decision-making, bottom-up ideation, and employee autonomy to foster an engaged and adaptable workforce [2, 3, 36, 46].

6.5 Discussion takeaway

This study, conceived as the first step of a community-led effort, is in line with the agile philosophy, promoting cooperation, adaptation, and ongoing improvement to collaboratively tackle the difficulties presented by OD.

In summary, OD can be defined as the comprehensive and nuanced characterization resulting from the synthesis of 13 diverse definitions of OD collected from the literature; "Organizational debt is the accumulation of outdated structures, policies, and processes that are no longer advantageous for the organization. Consequently, obstructs the advancement of the organization and its capacity to adjust to evolving conditions, ultimately hindering its potential for excellence.”

Manifestations and Impact of OD: Arising from the dual forces of obsolescence and accumulation, OD shares discernible parallels with process debt [19, 81]. The deleterious impact of OD resonates within the workforce, manifesting as a detriment to employee morale and performance. Developers, burdened by legacy issues stemming from OD, grapple with disengagement, hindering their ability to focus on innovation and creative problem-solving. Embracing the core tenets of the Agile Manifesto becomes imperative to infuse flexibility and adaptability, averting the pitfalls of superficial or ’fake’ agile implementations that may lead to escalated costs, delays, and pervasive frustration.

Mitigation Strategies for OD: Effectively navigating the labyrinth of OD mitigation mandates a holistic approach. Proactive identification and mitigation of OD sources emerge as linchpins for bolstering adaptability, efficiency, and sustained success. Acknowledging the inevitability of debt accumulation during organizational growth, leaders assume the role of diagnosticians, continuously evaluating areas where obsolescence or accumulation begets challenges [24, 33, 34]. The evaluative lens extends to the organizational structure, demanding continuous refinement to ensure alignment with strategic objectives and responsiveness to change. Adopting established TD quantification techniques represents a starting point for addressing OD. However, the social and organizational complexities of OD may require customized mitigation practices based on contextual needs.

Agile Mindset and Technological Adaptation: Championing an agile mindset serves as the catalyst for fostering resilience and innovation, propelling organizations to confront the ever-evolving business landscape. A flexible work environment, underpinned by a culture valuing innovation, experimentation and change, becomes pivotal. The cultivation of a continuous improvement ethos and the integration of feedback mechanisms empower employees, fortifying organizational agility. While performance, behavioral, and innovation metrics provide valuable insights into OD [42], their efficacy hinges upon contextual considerations within the agile paradigm.

Embracing Technological Advancements: Within the agile setting, embracing technological advancements emerges as a fundamental strategy for OD mitigation. Companies that stay updated on the latest technology and strategically incorporate them into their operations have simpler procedures and increased efficiency. This progressive strategy is in perfect harmony with agile principles, prioritizing quick adjustment to new tools and processes for continuous enhancement.

Optimizing Organizational Structures: It becomes crucial to prevent the accumulation of unneeded complications, which requires the optimization of organizational structures. An environment that fosters ongoing enhancement, along with a proactive attitude towards embracing change as a means of progress, establishes the foundation for a software company that is more adaptable and robust.

Future Directions and Community-Driven Initiatives: This innovative study, utilizing MLR approaches, examines the causes and effects of organizational dysfunction, represented as OD in software products and professional services. To tackle these difficulties, it is essential for academia and software firms to work together, following the agile philosophy of cross-functional teamwork. Adopting known TD methods [8, 9] serves as a fundamental approach to OD. Nevertheless, the distinct difficulties presented by OD necessitate the development of innovative approaches for its resolution. This study, conceived as the first step of a community-led effort, is in line with the agile philosophy, promoting cooperation, adaptation, and ongoing improvement to collaboratively tackle the difficulties presented by OD.

7 Conclusion

This MLR offers a conceptual foundation for the nascent topic of OD in software engineering, synthesizing diverse perspectives from industry experts. The analysis reveals OD arises from the accumulation of outdated structures, policies, and processes that are misaligned with an organization’s strategic goals. Poor change management, siloed efforts, avoiding disruption, and resistance to input are identified as key debt drivers. Consequences encompass reduced innovation, agility, competitiveness, morale, and employee retention.

Mitigating OD necessitates embracing agile principles such as cross-functional teams, minimum viable processes, transparency, decentralization, and delivering value rapidly. Ongoing feedback mechanisms, participatory decision-making, and continuous alignment of systems to evolving needs are vital. However, the reliance on non-peer-reviewed practitioner sources raises validity concerns regarding the lack of rigorous empirical grounding. Further research across diverse organizational and team settings is critically needed to validate the suggested OD causes, impacts, metrics, and management techniques.

Developing practical solutions requires cooperative efforts between academic institutions and software companies. OD management can build on existing best practices for quantifying, prioritizing and repaying TD, while also accounting for the distinct socio-technical complexities of OD. Further studies should investigate measurable metrics, workforce effects like burnout and turnover, customer satisfaction impacts, project success/failure outcomes, economic costs, and tailored organizational interventions.

Addressing this emerging interdisciplinary research domain necessitates drawing from diverse fields including management science, organizational behavior, anthropology, network analysis, and software engineering. Unraveling the complexities of OD is the key to maintaining innovation in the face of complexity and uncertainty. This is because organizational agility and flexibility are becoming increasingly important for competitiveness. To effectively address this challenge, a synergistic partnership between the business world and the academic world is required.

By synthesizing practitioner and academic insights, this study provides a comprehensive understanding of OD, emphasizing its multidimensional nature and the need for holistic management approaches. Software organizations can leverage these findings to diagnose OD sources, mitigate adverse consequences, and cultivate an agile and innovative organizational culture, ultimately enhancing their competitiveness and long-term success.

The conclusions from this exploratory MLR establish a conceptual foundation and future research agenda for the novel phenomenon of OD. Rigorously validating the identified concepts in empirical studies across diverse settings remains an open avenue. Developing contextualized management strategies demands a human-centric approach considering both business performance and employee experience metrics. As software pervades all facets of life, understanding the obstacles posed by dysfunctional structures and policies is crucial for organizational excellence.

Supporting information

S1 Checklist. PRISMA checklist.

(DOCX)

pone.0308183.s001.docx (30.6KB, docx)

Data Availability

All relevant data are within the manuscript and its Supporting information files.

Funding Statement

This work was supported by the KK-stiftelsen, Sweden, through the NODLA project (20200253 to M.O.A) and Helge Ax: son Johnsons Stiftelse.

References

  • 1.Sharp H., Robinson H., Woodman M. 2000. Software engineering: community and culture. IEEE Softw. (2000). 17(1), 40–47. [Google Scholar]
  • 2.Al-Baik O., & Miller J. (2018). Integrative double kaizen loop (IDKL): towards a culture of continuous learning and sustainable improvements for software organizations. IEEE transactions on Software Engineering, 45(12), 1189–1210. [Google Scholar]
  • 3.Al-Baik, O., & Miller, J. (2016, January). Kaizen cookbook: The success recipe for continuous learning and improvements. In 2016 49th Hawaii International Conference on System Sciences (HICSS) (pp. 5388–5397). IEEE.
  • 4.Ahmad M.O., Dennehy D, Conboy K., & Oivo M. (2018). Kanban in software engineering: A systematic mapping study. Journal of Systems and Software, 137, 96–113. [Google Scholar]
  • 5.Ramasubbu S., Balan R.K., Krishnan S.M., Padmanabhan P.K. 2017. The analytics of organizational culture. Management Science, 64(7), 3077–3094. [Google Scholar]
  • 6.M.O. Ahmad, Kuvaja, P., Oivo, M., & Markkula, J. (2016). Transition of software maintenance teams from Scrum to Kanban. In 2016 49th Hawaii International Conference on System Sciences. pp. 5427–5436.
  • 7.P. Avgeriou, P. Kruchten, I. Ozkaya, and C. Seaman. 2016. Managing technical debt in software engineering (dagstuhl seminar 16162). In Dagstuhl reports. 6(4). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
  • 8.Rios N., de Mendonca Neto M.G., Spinola R.O. 2018. A tertiary study on technical debt: types, management strategies, research trends, and base information for practitioners. Information Software Technology. 2018; 102:117–145. doi: 10.1016/j.infsof.2018.05.010 [DOI] [Google Scholar]
  • 9.Khomyakov I., Makhmutov Z., Mirgalimova R., and Sillitti A., 2019. Automated measurement of technical debt: A systematic literature review. In International Conference on Enterprise Information Systems. 95–106. [Google Scholar]
  • 10.Ahmad M.O., Markkula J., and Oivo M. 2013. Kanban in software development: A systematic literature review. Euromicro conference on software engineering and advanced applications. IEEE. 9–16. [Google Scholar]
  • 11.Al-Baik O., & Miller J. (2014). Waste identification and elimination in information technology organizations. Empirical Software Engineering, 19, 2019–2061. [Google Scholar]
  • 12.W. Cunningham. 1992. The WyCash portfolio management system. Addendum to the Proceedings on Object-oriented Programming Systems, Languages, and Applications (Addendum), Vancouver, British Columbia, Canada, ACM. 1992.
  • 13.Lenarduzzi V., Besker T., Taibi D., Martini A., Fontana F.A. 2021. A systematic literature review on technical debt prioritization: strategies, processes, factors, and tools. Journal of Systems and Software. 2021; 171:110827. doi: 10.1016/j.jss.2020.110827 [DOI] [Google Scholar]
  • 14.H. Saeeda, M.O. Ahmad, and T. Gustavsson. 2023. Multivocal Literature Review on Non-Technical Debt in Software Development: An Exploratory Study. In 18th International Conference on Evaluation of Novel Approaches to Software Engineering, 89–101.
  • 15.Rios N., Spinola R.O., Mendonça M., and Seaman C. 2020. The practitioners’ point of view on the concept of technical debt and its causes and consequences: a design for a global family of industrial surveys and its first results from Brazil. Empirical Software Engineering, 25, 3216–3287. [Google Scholar]
  • 16.Avgeriou P. C., Taibi D., Ampatzoglou A., Fontana F.A., Besker T., Chatzigeorgiou A., et al. 2020. An overview and comparison of technical debt measurement tools. IEEE software, 38(3), 61–71.34017155 [Google Scholar]
  • 17.Behutiye W.N., Rodríguez P., Oivo M., and Tosun A. 2017. Analyzing the concept of technical debt in the context of agile software development: A systematic literature review. Information and Software Technology, 82, 139–158. [Google Scholar]
  • 18.Besker T., Martini A., and Bosch J. 2018. Managing architectural technical debt: A unified model and systematic literature review. Journal of Systems and Software, 135, 1–16. [Google Scholar]
  • 19.Ahmad M.O., and Gustavsson T. 2022. The Pandora’s Box of social, process, and people debts in software engineering. Journal of Software: Evolution and Process. e2516. doi: 10.1002/smr.2516 [DOI] [Google Scholar]
  • 20.Z. Liu, V. Stray, and T. Sporsem. 2023. Organizational Debt in Large-Scale Hybrid Agile Software Development: A Case Study on Coordination Mechanisms. To be appear in Lecture Notes in Business Information Processing (LNBIP, volume 489): Springer International Publishing. https://link.springer.com/book/9783031485497
  • 21.Garousi V., and Mantyla M.V. 2016. When and what to automate in software testing? A multi-vocal literature review. Information and Software Technology, 76, 92–117. [Google Scholar]
  • 22.Haddaway N. R., Collins A. M., Coughlin D., and Kirk S., 2015. The role of Google Scholar in evidence reviews and its applicability to grey literature searching, PloS one, vol. 10, no. 9, pp. e0138237. doi: 10.1371/journal.pone.0138237 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Jaap Trouw, 2021. Organizational debt, an analogy. www.linkedin.com/pulse/organizational-debt-analogy-jaap-trouw/
  • 24.Sergio Caredda 2022. Are you Repaying your Organizational Debt? https://sergiocaredda.eu/organization/organization-design/are-you-repaying-your-organizational-debt/&cd=8&hl=en&ct=clnk&gl=se
  • 25.Carlos Piqueres 2021. Managing debt: Organizational debt. https://carlos-piqueres.medium.com/managing-debt-organizational-debt-a7a1578235f3
  • 26.Dragan Jojic. 2016. The Agility Challenge. https://www.infoq.com/articles/agility-challenge/
  • 27.Keyvanakbary. 2020. An Elegant Puzzle: Systems of Engineering Management. https://github.com/keyvanakbary/learning-notes/blob/master/books/an-elegant-puzzle.md
  • 28.Olaf Lewitz. 2015. Organizational Debt Cycle. https://trustartist.com/2015/01/15/organizational-debt-cycle/
  • 29.Tomasz Woźniak. 2019. Business in the digital age. https://www.futuremind.com/insights/business-digital-age
  • 30.Aaron Dignan. 2020. Brave New Work. https://antulik.com/2020-04-12-brave-new-work.html
  • 31.Deloitte. 2022. How to recognize the symptoms of technical and organizational debt? https://www2.deloitte.com/nl/nl/pages/human-capital/articles/how-to-recognize-the-symptoms-of-technical-and-organizational-debt.html
  • 32.Aaron Dignan. 2016. How to Eliminate Organizational Debt. https://medium.com/the-ready/how-to-eliminate-organizational-debt-8a949c06b61b
  • 33.Scott Belsky. 2016. Avoiding Organizational Debt. https://medium.com/positiveslope/avoiding-organizational-debt-3e47760803a0
  • 34.Ben Horowitz. 2012. Management Debt. https://a16z.com/2012/01/19/management-debt/
  • 35.Steve blank. 2015. Organizational Debt is like Technical debt–but worse. https://steveblank.com/2015/05/19/organizational-debt-is-like-technical-debt-but-worse/
  • 36.Linda Parker Gates. 2017. Are We Creating Organizational Debt? https://insights.sei.cmu.edu/blog/are-we-creating-organizational-debt/
  • 37.Bauer, D. 2019. On the importance of reducing organizational debt—And how we did it. NZZ Open. https://medium.com/nzz-open/on-the-importance-of-reducing-organizational-debt-and-how-we-did-it-7117b4c03f72
  • 38.Ben-Yosef, A. 2020. Lowering Org Debt: Spotting Org Smells. Medium. https://avivby.medium.com/lowering-org-debt-spotting-org-smells-8703fa1ef788
  • 39.Casasola, T. 2016. The Unexpected Psychology of Organizational Debt. The Ready. https://medium.com/the-ready/the-unexpected-psychology-of-organizational-debt-61ac89b4795b
  • 40.Duagi, B. 2019. It’s time to start tackling organizational debt. www.linkedin.com/pulse/its-time-start-tackling-organizational-debt-b%C3%BClent-duagi/
  • 41.Starrenburg, S. 2021. Why ‘Organizational Debt’ is Such a Powerful Concept.https://adeliberatelife.org/organizational-debt-powerful-concept/
  • 42.Stefan. L. 2023. The Hidden Threat to Your Organization: Organizational Debt Syndrome. https://www.linkedin.com/pulse/hidden-threat-your-organization-organizational-debt-stefan-lindegaard/
  • 43.Lorenz Solothurnmann—Organizational debt. www.linkedin.com/posts/solothurnmann_organizationaldebt-businessagility-inpositiv-activity-7071382812171382785-41p1/?originalSubdomain=lb
  • 44.Steve Blank and Pete Newell (2017). What Your Innovation Process Should Look Like. https://hbr.org/2017/09/what-your-innovation-process-should-look-like
  • 45.Jarvis WF. Organizational debt levels: harbinger of change? Among healthcare organizations, debt has increased steadily in recent years. Now, most are deleveraging—perhaps in response to less favorable operating trends. Healthc Financ Manage. 2003;57(2):56–61. [Google Scholar]
  • 46.Liu Z, Stray V, Sporsem T. Organizational Debt in Large-Scale Hybrid Agile Software Development: A Case Study on Coordination Mechanisms. In: Honious S, Cuomo D, Ralyté J, editors. Advanced Information Systems Engineering Workshops. Cham: Springer; 2023. p. 130–145. [Google Scholar]
  • 47.Schultz C, Salomo S, Brentani U, Kleinschmidt EJ. How formal control influences decision-making clarity and innovation performance. J Prod Innov Manag. 2013;30(3):430–447. [Google Scholar]
  • 48.Andriole SJ. Managing organizational debt: Understanding and addressing liabilities. American Productivity & Quality Center (APQC); 2020.
  • 49.De Smet A, Kleinmann M, Lund S, Tuttle R. Unhealthy patterns: How to identify and overcome them. McKinsey Quarterly. 2022;(3):72–85. [Google Scholar]
  • 50.Perlow LA. Sleeping with your smartphone: How professional norms undermine workplace unplugging. MIT Sloan Management Review. 2022;63(3):27–35. [Google Scholar]
  • 51.Galbraith JR. Organizing to deliver solutions. Organ Dyn. 2002;31(2):194–207. [Google Scholar]
  • 52.Syverson C. What determines productivity?. J Econ Lit. 2011;49(2):326–65. [Google Scholar]
  • 53.Tarafdar M, Pullins EB, Ragu-Nathan TS. Technostress: negative effect on performance and possible mitigations. Inf Syst J. 2015;25(2):103–132. [Google Scholar]
  • 54.H. Krasner. 2021. The cost of poor software quality in the US: A 2020 report. Published Online. www.it-cisq.org/pdf/cpsq-2020-report.pdf
  • 55.Sweden. 2019. The Swedish National Audit: Föråldrade it-system–Hinder för en effektiv digitalisering
  • 56.M. Senge. 2006. The fifth discipline: The art and practice of the learning organization. Broadway Business.
  • 57.Edmondson A. 1999. Psychological safety and learning behavior in work teams. Administrative science quarterly, 44(2), 350–383. [Google Scholar]
  • 58.Nadler D.A., Tushman M.L. 1980. A model for diagnosing organizational behavior. Organizational dynamics, 9(2), 35–51. [Google Scholar]
  • 59.Luthans F., Youssef-Morgan S.M. 2017. Psychological capital: An evidence-based positive approach. Annual Review of Organizational Psychology and Organizational Behavior, 4, 339–366. [Google Scholar]
  • 60.Maslach C., Schaufeli W.B., Leiter M.P. 2001. Job burnout. Annual review of psychology, 52(1), 397–422. doi: 10.1146/annurev.psych.52.1.397 [DOI] [PubMed] [Google Scholar]
  • 61.R.T. Mowday, L.W. Porter, R.M. Steers. 2013. Employee—organization linkages: The psychology of commitment, absenteeism, and turnover. Academic press.
  • 62.Lepine J.A., Podsakoff N.P., Lepine M.A. 2005. A meta-analytic test of the challenge stressor–hindrance stressor framework: An explanation for inconsistent relationships among stressors and performance. Academy of Management Journal, 48(5), 764–775. [Google Scholar]
  • 63.Anderson E.W., Sullivan M.W. 1993. The antecedents and consequences of customer satisfaction for firms. Marketing science, 12(2), 125–143. [Google Scholar]
  • 64.Delone W.H., McLean E.R. 2003. The DeLone and McLean model of information systems success: a ten-year update. Journal of management information systems, 19(4), 9–30. [Google Scholar]
  • 65.Boehm B.W. 1991. Software risk management: principles and practices. IEEE software, 8(1), 32–41. [Google Scholar]
  • 66.L.C. Briand, K. El Emam, F. Bomarius. 1998. COBRA: a hybrid method for software cost estimation, benchmarking, and risk assessment. In Proceedings of the 20th international conference on Software engineering. 390–399.
  • 67.D.N. Card. 1991. MEASURING SOFTWARE DESIGN QUALITY. Prentice-Hall, Inc., Upper Saddle River, NJ (United States).
  • 68.Kemerer C.F., Slaughter S. 1999. An empirical approach to studying software evolution. IEEE Transactions on Software Engineering, 25(4), 493–509. [Google Scholar]
  • 69.T. Klinger, P. Tarr, P. Wagstrom, C. Williams. 2011. An enterprise perspective on technical debt. In Proceedings of the 2nd Workshop on managing technical debt. 35–38.
  • 70.N.A. Ernst, S. Bellomo, I. Ozkaya, R.L. Nord, I. Gorton. 2015. Measure it? Manage it? Ignore it? Software practitioners and technical debt. In Proceedings of the 2015 10th joint meeting on foundations of software engineering. 50–60.
  • 71.Kruchten P., Nord R.L., Ozkaya I. 2012. Technical debt: From metaphor to theory and practice. IEEE software, 29(6), 18–21. [Google Scholar]
  • 72.Li Z., Avgeriou P., Liang P. 2015. A systematic mapping study on technical debt and its management. Journal of systems and software, 101, 193–220. [Google Scholar]
  • 73.A. Martini, J. Bosch. 2015. The danger of architectural technical debt: Contagious debt and vicious circles. In 2015 12th Working IEEE/IFIP Conference on Software Architecture (WICSA). 1–10.
  • 74.Morgeson F.P., Hofmann D.A. 1999. The structure and function of collective constructs: Implications for multilevel research and theory development. Academy of management review, 24(2), 249–265. [Google Scholar]
  • 75.K. Barton, D. Freiling. 2016. What’s your organizational debt? And why it matters. Bain & Company.
  • 76.Blank S. 2021. Building a Company to Thrive in Uncertainty. Harvard Business Review, 99(5), 84–93. [Google Scholar]
  • 77.Wasserman S. 1994. Social network analysis in the social and behavioral sciences. Social network analysis: Methods and applications, 1, 1–27. [Google Scholar]
  • 78.Rainer A., Hall T. 2003. Key success factors for implementing software process improvement: a maturity-based analysis. Journal of Systems and Software, 66(2), 71–84. [Google Scholar]
  • 79.Dodgson M., Gann D.M., Phillips N. 2013. Organizational learning and the technology of foolishness: The case of virtual worlds at IBM. Organization Science, 24(5), 1358–1376. [Google Scholar]
  • 80.C.E. Helfat, S. Finkelstein, W. Mitchell, M.A. Peteraf, H. Singh, D.J. Teece, S.G. Winter. 2007. Dynamic capabilities: Understanding strategic change in organizations. John Wiley & Sons.
  • 81.A. Martini, T. Besker, and J. Bosch. 2020. Process debt: a first exploration. In 27th Asia-Pacific Software Engineering Conference. IEEE. 316–325.

Decision Letter 0

Vanessa Carels

22 May 2024

PONE-D-24-05123Organizational Debt – Roadblock to Agility in Software Engineering: Exploring an Emerging Concept and Future Research for Software ExcellencePLOS ONE

Dear Dr. Al-Baik,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please note that we have only been able to secure a single reviewer to assess your manuscript. We are issuing a decision on your manuscript at this point to prevent further delays in the evaluation of your manuscript. Please be aware that the editor who handles your revised manuscript might find it necessary to invite additional reviewers to assess this work once the revised manuscript is submitted. However, we will aim to proceed on the basis of this single review if possible. 

Please submit your revised manuscript by Jul 05 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Vanessa Carels

Staff Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that your Data Availability Statement is currently as follows: All relevant data are within the manuscript and its Supporting Information files.

Please confirm at this time whether or not your submission contains all raw data required to replicate the results of your study. Authors must share the “minimal data set” for their submission. PLOS defines the minimal data set to consist of the data required to replicate all study findings reported in the article, as well as related metadata and methods (https://journals.plos.org/plosone/s/data-availability#loc-minimal-data-set-definition).

For example, authors should submit the following data:

- The values behind the means, standard deviations and other measures reported;

- The values used to build graphs;

- The points extracted from images for analysis.

Authors do not need to submit their entire data set if only a portion of the data was used in the reported study.

If your submission does not contain these data, please either upload them as Supporting Information files or deposit them to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of recommended repositories, please see https://journals.plos.org/plosone/s/recommended-repositories.

If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent. If data are owned by a third party, please indicate how others may request data access.

3. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Overall, the study addresses an interesting topic that worth investigating. It well explained the motivation and purpose for the study, and use the right methodology to approach the study.

However, the main limitation of the study is the lack of data to sufficiently ground the findings. The data is only contain a few amount of blogs (22 blogs) with no idea how intense each blog is. This limit amount of data might severely undermine the quality of the results. I would suggest the authors to augment the data with some related research papers about OD. You might need to go beyond software engineering research avenue (checking research avenue in the management and organization area) , examples are:

- William F. Jarvis. "Organizational debt levels: harbinger of change? Among healthcare organizations, debt has increased steadily in recent years. Now, most are deleveraging--perhaps in response to less favorable operating trends"

- Liu, Zixuan, Viktoria Stray, and Tor Sporsem. "Organizational Debt in Large-Scale Hybrid Agile Software Development: A Case Study on Coordination Mechanisms."

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2024 Nov 25;19(11):e0308183. doi: 10.1371/journal.pone.0308183.r002

Author response to Decision Letter 0


26 Jun 2024

Dear Editors and Reviewers,

Thank you for your valuable feedback and suggestions on our manuscript titled "Organizational Debt in Software Engineering: A Multivocal Literature Review." We appreciate the time and effort you have invested in reviewing our work and providing constructive comments to improve its quality.

We have carefully considered the points raised by the reviewer and would like to address them in this response letter.

Regarding the concern about the lack of data to sufficiently ground the findings, we acknowledge the limitation of relying solely on blog posts and practitioner literature. As mentioned in the paper, the topic of organizational debt (OD) is an emerging concept in software engineering, and there is a scarcity of academic literature on this subject. Our motivation for conducting a multivocal literature review (MLR) was to synthesize the perspectives and experiences of industry professionals, who are at the forefront of grappling with OD challenges.

However, we recognize the importance of complementing these practitioner insights with academic research to strengthen the validity and rigor of our findings. We thank the reviewer for suggesting the paper by William F. Jarvis and the case study by Zixuan Liu et al. These sources provide valuable insights from the management and organization domains, which can enrich our understanding of OD in the software engineering context.

To address this concern, we propose the following revisions to our manuscript:

1. We conducted a thorough search for relevant academic literature on OD from related disciplines, such as management, organizational behavior, and information systems. This has broaden our data corpus and has provided a more comprehensive view of the phenomenon. Please, see pages 4-6

2. We incorporated the suggested papers by William F. Jarvis and Zixuan Liu et al., as well as eight (8) other relevant academic sources, into our analysis. This allows us to triangulate the practitioner perspectives with theoretical and empirical findings from academic research. Please, see results section on pages 5-6.

3. In the revised manuscript, we clearly distinguish between insights derived from practitioner literature and those from academic sources. This will hopefuly enhance transparency and will allow the reader to assess the credibility and generalizability of our findings. Please, see results section on pages 5-6.

4. We adjusted the methodology section to reflect the inclusion of academic literature and provide a detailed description of the search strategy, selection criteria, and analysis process for these additional sources. Please, see research method section on pages 4-5.

5. The discussion section has been expanded to critically evaluate the convergence or divergence of practitioner and academic perspectives on OD, as well as the implications for software engineering research and practice. Please, see discussion section on pages 13-14.

We believe that addressing the reviewer's concern by incorporating relevant academic literature has strengthen our study's validity and contribute to a more comprehensive understanding of the organizational debt phenomenon in software engineering.

Thank you again for your valuable feedback and the opportunity to improve our work. We look forward to addressing these concerns and submitting a revised manuscript for your consideration.

Sincerely,

On behalf of the authors,

Osama Al-Baik

Attachment

Submitted filename: Response to Reviewers.docx

pone.0308183.s002.docx (13.5KB, docx)

Decision Letter 1

Leander Luiz Klein

18 Jul 2024

Organizational Debt – Roadblock to Agility in Software Engineering: Exploring an Emerging Concept and Future Research for Software Excellence

PONE-D-24-05123R1

Dear Dr. Osama Al-Baik,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Leander Luiz Klein, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Dear authors!

Thank you for making the revision of the article following the reviewer' suggestion. I send the article to the editor in chief with my final decision.

Best regards.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Abdullah Aldaeej

**********

Acceptance letter

Leander Luiz Klein

28 Oct 2024

PONE-D-24-05123R1

PLOS ONE

Dear Dr. Al-Baik,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Professor Leander Luiz Klein

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Checklist. PRISMA checklist.

    (DOCX)

    pone.0308183.s001.docx (30.6KB, docx)
    Attachment

    Submitted filename: Response to Reviewers.docx

    pone.0308183.s002.docx (13.5KB, docx)

    Data Availability Statement

    All relevant data are within the manuscript and its Supporting information files.


    Articles from PLOS ONE are provided here courtesy of PLOS

    RESOURCES