Models and methods for information systems project success evaluation – A review and directions for research

Abstract

Organizations heavily rely on information systems to improve their efficiency and effectiveness. However, on the one hand, information systems projects have often been seen as problematic endeavors. On the other hand, one can ask if this perspective results from subjective perceptions or objective assessments. We cannot find a definitive answer to this question in the literature. Moreover, there is no structured information about the models and methods currently available to assess projects' success in practice. This paper aims to present the results of a literature review carried out on the extant models and methods for evaluating the success of information systems projects. Additionally, it also offers models and methods from other areas that may be suitable for assessing IS projects. Results show that most models and methods found in the literature are, in their essence, theoretical exercises with only a few pieces of evidence of their use in practice, thus urging for more empirically based research.

Project evaluation; Project assessment; Project appraisal; Project evaluation models; Project evaluation methods; Project evaluation techniques; Evaluation criteria; Success criteria; Project success; Information systems; Project management.

1. Introduction

Information Systems (IS) are crucial for contemporary organizations, being present in all business aspects (Varajão and Carvalho, 2018; Kääriäinen et al., 2020). In today’s VUCA (Volatile, Uncertain, Complex, and Ambiguous) world, the capacity to maintain and update existing IS and create and adopt new IS features is a competitive differentiator (Patnayakuni and Ruppel, 2010; Ngereja and Hussein, 2021). Furthermore, organizations need to innovate in products, processes, markets, and business models to remain sustainable and perform effectively (David and Lawrence, 2010). Without IS projects, such as digital transformation projects, which refer to the changes in working and business offerings enabled by the adoption of Information Technologies (IT) in an organization (Kääriäinen et al., 2020), that is not viable.

In the last decades, project management has gained recognition as an academic discipline because all organizations develop projects and, for this, resort to project management as a way of structuring and managing their investments. However, in the particular case of IS, projects continue to report lower levels of success (Iriarte and Bayona, 2020), so it is crucial to understand the influencers of project success (Kendra and Taplin, 2004) and how to evaluate it (Varajão, 2016, 2018; Pereira et al., 2022). Additionally, it is often thought that achieving success in these projects is challenging (Tam et al., 2020).

Evaluation of success is concerned with judgments about the achievements of an endeavor (Arviansyah et al., 2015; Pereira et al., 2022), and appropriate methods should be adopted for evaluating projects (Pinto and Slevin, 1987). The success of projects has been traditionally related to the Iron Triangle, i.e., to the accomplishment of scope, cost and time. More recently, other important criteria have been considered in the evaluation of success, such as stakeholders' satisfaction or business impact (Varajão et al., 2021).

On the “dark side”, there are studies that reveal high levels of failure, as is the case, for instance, of a global study of IT change initiatives covering 1471 projects, that concludes that one out of six projects ran, on average, 200% over budget and 70% over schedule (Hoang et al., 2013); several authors, such as Cecez-Kecmanovic and Nagm (2008) or Iriarte and Bayona (2020), also mention such disappointing success rates. As expected, this uncertainty of value realization troubles both practitioners and researchers.

Conversely, on the “bright side”, some researchers and practitioners (e.g., Lech (2013)) have been questioning these numbers because the world around us is full of useful and reliable (successful) IT applications, which debunks that negative perspective of success (Varajão et al., 2022b). The lack of details (namely, regarding samples or criteria) of most studies also helps raise criticism regarding the reported results (Sauer et al., 2007). Furthermore, in most studies, it is impossible to ascertain the methods or techniques used for evaluating and reporting the success of IS projects or whether (and how) projects are formally assessed in practice. Some studies addressing the evaluation of project success in practice support this concern (Pereira et al., 2022; Varajão and Carvalho, 2018).

The rapid advance in IT has boosted IS project development in organizations for reorganizing businesses and improving services. Such projects help organizations create and maintain competitive advantages through fast business transactions, increasingly automated business processes, improved customer service, and adequate decision support. Considering that any organization’s sustainable success is strongly associated with IS success and, consequently, with the success of IS projects, evaluating these projects assumes critical importance in modern organizations (Ma et al., 2013; Pereira et al., 2022).

Although “success” is a frequently discussed topic, a consensus concerning its meaning is rarely reached. This is due to the fact that project success is an intricate and elusive concept with several different meanings (McCoy, 1986; Thomas and Fernández, 2008). In effect, the concept of success can have several interpretations because of the different perceptions it generates, leading to disagreements about what can be considered a successful project (Baccarini, 1999). Since the 1950s, many authors have accepted the triple constraints (time, cost, specification) as a standard measure of success (Oisen, 1971; Atkinson, 1999). These continue to be very important in evaluating the success of IS projects, together with other criteria such as stakeholders' satisfaction or project benefits (Varajão and Carvalho, 2018). However, an IS project cannot always be seen as a complete success or a complete failure, and different stakeholders may perceive the terms “success” and “failure” differently (Milis and Vanhoof, 2006). To make things even more challenging regarding evaluation, organizational effectiveness is paradoxical (Cameron, 1986), and projects have priority, structural, and execution tensions (Iivari, 2021).

Although there are several models, methods, and techniques to evaluate projects' success, the lack of structured information about them (e.g., characteristics, context, or results achieved in practice) may hinder their use by practitioners. Without such information, it can be quite difficult to identify which models or methods are adequate to evaluate a project’s success, considering the implementation feasibility, benefits, and limitations of each alternative. It also makes it difficult for researchers to identify opportunities for new contributions.

The evaluation of project success seems to be currently an informal and rudimentary process based on perceptions, mainly focused on project management’s success and not concerned with the success of the projects' outputs (Varajão and Carvalho, 2018; Pereira et al., 2022). In other words, many times, success is not formally evaluated, and even when the evaluation is carried out, it is based on an incomplete set of criteria or limited evaluation models. The consequences of not formally evaluating the success of a project may result in the waste of efforts and resources (Pujari and Seetharam, 2015) and misperception of results (Turner and Zolin, 2012).

Furthermore, as aforementioned, it is interesting to note that several studies report on project success, but not much is known about how it is being evaluated concerning the techniques or methods used by project managers and organizations. For instance, the Standish Group’s studies (e.g., Standish (2018); StandishGroup (2020)) and other studies (e.g., Marnewick (2012)) declare the success achieved in projects. Still, there is no information about “whether” and “how” the formal success assessment was carried out in practice by the participants (often, only a few criteria are mentioned, and the reported success is based on “perceptions” of the study’s participants).

Aiming to fill this gap in the literature as well as improve awareness about the currently available models and methods for evaluating IS projects' success, we conducted a literature review. The purpose is to summarize the extant research, providing a framework of models and methods for researchers and practitioners that identifies their characteristics, underlying techniques, contexts of application, benefits, limitations, and empirical support. We also present a review of other project areas in order to have richer results since some models and methods are not dependent on the project type. Moreover, the discussion includes main insights and future directions for research.

The structure of the article is as follows: Section 2 presents the main concepts; Section 3 addresses the research questions and the research method; Section 4 presents the results; Section 5 discusses the main findings; and, finally, Section 6 addresses the contributions and limitations.

2. Main concepts

Our study focuses on models and methods for evaluating IS project success. To enable a comprehensive understanding of the subject, as depicted in Figure 1, there are several related concepts that are important to clarify. The concepts of “Information Systems projects”, “success of projects”, and “evaluation of success” are presented in the next subsections. The models and methods for project success evaluation are presented in Section 4 and discussed in Section 5.

Figure 1.

Conceptual framework.

2.1. Information systems projects

A project is an undertaking to create something that does not exist yet, which needs to be delivered in a set time and at an agreed cost. IS projects include all the common characteristics of other projects. However, they also have particularities, such as providing a service to implement IT solutions, eventually including the assessment of the project outcome (Kutsch and Maylor, 2009).

In other words, IS projects are temporary endeavors that lead to some unique outputs and outcomes related to IT adoption. These outputs and outcomes can be, for instance, changes in business processes; the renewal of the IT infrastructure; the adoption of new software applications; etc. On the one hand, sometimes, the success of the outcome/project can be assessed right after it is delivered (Varajão et al., 2022a). On the other hand, many times, a complete account of the project’s success can only be obtained long after the end of the project; in other words, after the impact of the outputs and outcomes on the enterprise is felt (Varajão and Carvalho, 2018).

2.2. Success of projects

One of the problems that show up frequently concerning the project’s success has its roots in the definition of “success”. The success of a project can be understood in diverse ways according to different stakeholders. On the one hand, time, costs and scope compliance are essential elements for a project’s success; on the other hand, the stakeholders' satisfaction or the achievement of business benefits play a prominent role. Therefore, the main concern should be meeting the client’s real needs (Paiva et al., 2011) since projects are typically designed to obtain benefits for the organization according to business objectives and value concerns (Keeys and Huemann, 2017).

In project management and over the years, the concept of success has undergone some significant changes. In the 1970’s, the success of a project was mainly focused on the operational dimension; the focus on the customer was practically non-existent. Since project management began to be a body of knowledge in the mid-twentieth century, many processes, techniques, and tools have been developed (Davis, 2014). Today, they cover various aspects of the project lifecycle and have made it possible to increase its efficiency and effectiveness (Varajão, 2016). According to Kerzner (2017), a more modern perspective assesses success in terms of primary (timely, within budget, and with the expected quality) and secondary aspects (customer acceptance and customer agreement regarding the use of his name as a reference). For De Wit (1988), a distinction must be made between project success and project management success in any discussion of success, bearing in mind that good project management can contribute to project success. According to Baccarini (1999), project management success is mainly related to the achievement of the project regarding scope, time, and cost, which indicate the efficiency and effectiveness of project execution. Usually, project management success can be evaluated at the end of a project. The success of the output is related to the impacts of the project resulting product(s)/service(s)/other on the business (e.g., business benefits), and its evaluation may only be possible later, at the post-project.

Cuellar (2010) states that project success can be considered objective when represented by measurable constructs such as time, schedule, and scope, or subjective if evaluated based on stakeholders' opinions.

2.3. Evaluation of success

Different meanings of assessment have also been presented throughout the years. For instance, APHA (1960) characterized assessment as “the process of determining the value or amount of success in achieving a predetermined objective”. Scriven (1991: 139) defines assessment as “the process of determining the merit, worth or value of something”. DAC (2002) has characterized assessment as “a precise and target appraisal of a continuous or finished task, program or approach, its plan, execution, and results”. Patton (1996: 14) describes program assessment as “the systematic collection of information about the activities, characteristics, and outcomes of programs for use by specific people to reduce uncertainties, improve effectiveness, and make decisions concerning what those programs are doing and affecting”. These definitions reflect ex-ante, observing, mid-term, and final assessments.

We can also add the ex-post assessment, which can be described as an assessment that is made after an intervention has been finished. In other words, ex-post assessment is directed after a specific period following the fulfillment of an undertaking, with emphasis on the adequacy and supportability of the task (Zidane et al., 2016).

Project success is a multi-dimensional concept that requires appropriate evaluation models and methods, which can be defined as practical tools used to measure the success of a project (Silvius and Schipper, 2015).

3. Method

The research method is presented in the following sections.

3.1. Literature review

Literature reviews aim to address some problems by identifying, critically evaluating, and integrating the findings of all relevant, high-quality individual studies addressing one or more research questions. A literature review might achieve all or most of the following objectives (Baumeister and Leary, 1997): establish to what extent existing research has progressed towards clarifying a particular problem; identify connections, contradictions, gaps, and inconsistencies in the literature, as well as explore reasons for these; formulate general statements or an overarching conceptualization (Sternberg, 1991); comment on, evaluate, extend, or develop theory; describe directions for future research. By doing this, implications for future practice and policy should be provided.

The literature review process is defined as an examination of a clearly formulated question (or questions) that uses systematic and explicit methods to identify, select, and critically appraise relevant research and to collect and analyze data from the studies that are included in the review (Cochrane, 2005).

The research started with problem formulation by defining the research questions. This was followed by the definition of data sources and search strategy. Then, a literature search was carried out in the selected database. After obtaining the results, an eligibility test was performed to identify candidate publications. The final set of publications was then selected after a quality assessment considering the inclusion and exclusion criteria. After getting all the relevant publications, the final steps involved data extraction, analysis, and interpretation. The remaining sections describe the research process in detail.

3.2. Research questions

By looking at the literature, it is easy to understand the difficulty of evaluating the success of a project, not only due to the subjective nature of the definition of success but also to the different characteristics, context, or complexity of projects and different ways of evaluating them. For each type of project, several evaluation models and methods can be applied. Therefore, it can be quite challenging to identify which models or methods are adequate to evaluate a project’s success.

The application of a model or method to evaluate the success should follow a well-justified process, considering the type and characteristics of each project in particular and the purpose of the evaluation. Even though several studies found in the literature focus on various aspects of project success, few studies address the evaluation process (Varajão and Trigo, 2016; Varajão, 2018; Pereira et al., 2022) and their respective models and methods.

To find out what is the state-of-the-art and create a framework of models and methods for evaluating the success of IS projects, we formulated the following primary research question:

RQ1

What are the models and methods for evaluating the success of IS projects currently available in the literature?

Since the models and methods used in other project areas may also be suitable to be used in IS projects, we formulated a secondary question:

RQ2

What are the models and methods for evaluating the success of non-IS projects currently available in the literature?

3.3. Data sources, search strategy and article selection

We decided to concentrate the search on the well-known database Elsevier’s Scopus (www.scopus.com) due to its wide coverage of scientific outlets. We are well aware that there are other databases or search engines that may contain relevant articles. However, the selected database includes major journals and conferences from the IS and project management areas.

Since the terminology used in published studies is much diversified, the use of synonyms and word variations has been required. Hence, when conducting the literature search, the following terms and synonyms were used:

• −Evaluat∗ (evaluation and evaluating), assess∗ (assessment and assessing), apprais∗ (appraisal, appraising), valuat∗ (valuation and valuating), estimat∗ (estimation and estimating), calculat∗ (calculation and calculating);
• −Performance, success, attainment, accomplishment, achievement, realization, realisation;
• −Method∗ (method and methods), technique∗ (technique and techniques), system∗ (system and systems), procedure∗ (procedure and procedures), process∗ (process and processes);
• −Project∗ (project and projects);
• −Information system∗ (information system and information systems), information technolog∗ (information technology and information technologies), information and communications technolog∗ (information and communications technology and information and communications technologies), IT/IS, IS/IT, ICT.

The search strings were formulated using logical expressions created from these terms. The total number of logically different expressions was 15. The search strings are listed in the appendix. Each search string’s logical structure was written according to the specific query format of the search engine.

We decided to carry out several searches, from more open searches to more restrictive ones, to guarantee at most that every relevant study was identified. Figure 2 synthesizes the searches performed and article selection results, including the search results and candidate publications per search expression (expn). The respective search expressions can be found in the appendix. It is important to note that, in some searches, we decided not to confine to IS projects to get answers to the secondary research question.

Figure 2.

Systematic Literature Review search and selection breakdown.

Searches were completed by the end of January 2021. In each search was downloaded a CSV file with all the results. These files were then compiled into one single file to remove duplicates and identify candidate publications that could have some connection/correlation with the subject. The searches resulted in the identification of 755 unique references.

The articles were selected for additional analysis mainly based on their title, abstract, and keywords. The abstract was read in order to verify if the title explicitly mentioned models or methods for project success evaluation. When the abstract did not provide enough information, the full article was read to assess its relevance. Some articles were later excluded because their content did not correspond to what was described in the title or abstract.

The inclusion criteria of sources were as follows: to present a model or a method for evaluating the success of projects; published in an academic journal or a conference; written in English. In turn, the exclusion criteria were the following: not written in English; published as a preface, editorial, article summary, interview, workshop, panel, or poster. To be considered, a source should comply with all the criteria.

As in Savolainen et al. (2012), the first selection was performed by one author and randomly checked by the others, resulting in 159 articles that could contain important information for the study. The second selection, made by two authors, was based on the full reading of the articles resulting from the first selection and led to the final set of 34 articles. These are the only articles related to techniques for evaluating projects (of IS and other areas). A significant difference is noticeable between the primary search results and the relevant results because we decided to have more open searches at first to avoid overlooking important references. This increased the article selection process effort but also increased confidence in the final results.

3.4. Data extraction and synthesis

The following data elements were extracted from the selected articles: model/method identification/name; underlying techniques/approaches; benefits (as presented by the authors); limitations/further developments (as presented by the authors); project types where the model/method was applied (e.g., information systems) and sample; and references.

It is noteworthy that we tried to be as objective as possible in the extraction and presentation of data. Good examples of this are the presented benefits and limitations since they are the ones stated by the authors (it is out of the scope of this article to carry out independent experimentation of each model/method, but it shapes an interesting path for further research).

A summary of the identified models/methods is presented in Section 4.

3.5. Validity of the literature review

For the validity assessment, we adopted the same procedure as Savolainen et al. (2012), which is described next.

3.5.1. Construct validity

The validity of the review is based on the assumption that the researchers authors of this study and the authors of the reviewed articles have a common understanding of the concepts presented in Section 2.

3.5.2. Internal validity

The review’s internal validity is assured by the procedure used for the articles' search, selection, and subsequent analysis. One primary threat to the validity of the reasoning used in an analysis may arise from the subjective evaluation of the articles since the evaluation results depend solely on the evaluator. In this study, the evaluation procedure was predefined and approved by two researchers to make the reasoning valid and repeatable. The review’s internal validity has been ensured by the review procedure documentation and the complete analysis verification. Therefore, the threats to internal validity are not significant in this case.

3.5.3. Repeatability

The search for articles was performed systematically and can be easily repeated using the queries presented in the appendix. However, a major threat to this literature review’s repeatability is that it is based on search engine results. The results of new searches may not be exactly identical due to the constantly growing nature of the databases. The publishers frequently add new articles to the databases and may add old articles to them in some cases as well. Hence, new articles may appear in future search results. Nevertheless, that is also not expected, at least at a significant level. Searches might also have missed relevant articles. Since we included synonyms in the search strings and two of the researchers double-checked the results, this risk was reduced to a minimum.

3.5.4. Article selection and analysis

The first selection of articles was partially cross-checked. One researcher randomly checked the searches and article selection. This procedure avoided unnecessary errors. Two researchers performed the final selection and analysis of the articles.

4. Results

Answering the primary and secondary research questions, Tables 1 and 2 present, respectively, for IS projects and non-IS projects, the models/methods for evaluating the success of projects identified in the literature. For each model/method, the following is presented: identification/name; underlying techniques/approaches; benefits (as described by the authors); limitations/further developments (as described by the authors); project types where the model/method was applied, and sample; and references.

Table 1.

Benefits/limitations of models/methods for the evaluation of IS project success.

Model/Method	Uses/Applies/Is based on	Benefits (according to the authors)	Limitations/Further developments (according to the authors)	Project Types/Sample	References
Model for evaluating the performance of projects	Data Envelopment Analysis (DEA)	Handle multiple project inputs. Handle multiple project outputs. No assumption of a functional form relating inputs to outputs is necessary. Projects are directly compared against peer operational units or a combination of peer operational units. The model’s inputs and outputs can be in different operational units. It produces an efficiency score for projects. Enables linking an inefficient project to its benchmark efficient project(s). It enables ordering projects by efficiency. Enables to evaluate the relative performance of projects and make resource decisions.	Noisy data can cause significant problems. DEA is good at estimating the “relative” efficiency of the operational units, but it does not allow comparison to a theoretical maximum. Large problems can be computationally intensive. The research was limited to only security-based open-source software projects.	IT projects Sample: 34 open-source software projects (illustrative)	(Wray and Mathieu, 2008)
Methodology for project performance evaluation	Balanced Scorecard (BSC); Hesitant Fuzzy Set (HFS)	Integrates BSC and HFS to specify the performance metrics (to find weights to measure the performance in fuzzy environments), and most real-life applications include fuzziness. The critical success factors are determined based on expert judgments. HFS is used for a subjective weighting of project performance factors. Note: The outputs of the study were found acceptable by a company.	Future applications may include an awarding system.	IT projects Sample: 68 projects of a company (experimentation in practice)	(Basar, 2020)
Method for project management performance measurement	Balanced Scorecard (BSC); Fuzzy Analytical Network Processing (FANP)	Measure performance from a multi-domain perspective of project management. It enables multi-dimensional measurement. Enables to determine weighting priorities for key performance indicators. IT is an integrated method for qualitative and quantitative measurements.	No limitations were described by the authors.	IT/IS projects Sample: 1 SAP rollout project (illustrative)	(Hermawan et al., 2016)
Approach for evaluating project effectiveness	Fuzzy Logic Expert approach, based on Fuzzy Linguistic Parameters and Fuzzy values	It can be used for an impartial evaluation of developed software projects. Measure the success of a project but also keeps track of serious activities responsible for the achievement of the Six Sigma standard in the organization.	No limitations were described by the authors.	Software projects Sample: 10 software projects developed through Six Sigma methodology (illustrative)	(Pujari and Seetharam, 2015)
Project performance evaluation model	Deep Belief Networks (DBN)	The model automatically assigns weights and allocates ceiling scores to the project elements based on the DBN weights, which capture the interaction between the project elements. It does not depend on humans to manually assign weights. It eliminates much of the subjectivity inherent in manually assigning weights to criteria by automatically assigning and adjusting model weights during training and validation. DBN has the capability to automatically adapt to new knowledge even after the model becomes fully operational. It is more objective because there is a less human influence in the assigning of model weights. The DBN approach is universal because it takes into account the interaction between the project elements in the computation of the maximum scores as well as the monthly project performance indicator. DNB has the ability to store learned knowledge and make it available for use, as well as automatically update the knowledge base as new knowledge becomes available in the operation of the model.	No limitations were described by the authors.	IT projects Sample: 13 projects (illustrative)	(Nguvulu et al., 2012)
Project performance evaluation approach	Back Propagation Deep Belief Network (BP-DBN)	It eliminates the manual assignment of model weights. Automatically assigns weights to the project elements during model training, with the possibility of self-adjustment as new data is presented to the operational model. Enables to compute reliable monthly project performance indicators based on the basic project elements data. It increases the objectivity of the project evaluation process. It provides a more rational and objective evaluation approach.	No limitations were described by the authors.	IT projects Sample: 7 projects (illustrative)	(Nguvulu et al., 2011)
Project assessment model	Principal Component Analysis (PCA); Ordered Weighted Averaging (OWA)	The number of criteria is reduced by PCA but preserving the original information. The weights of resultant criteria are obtained by employing OWA. It reduces the complexity of software project evaluation. It provides the flexibility for deciding the weight distribution among criteria.	No limitations were described by the authors.	Software projects Sample: 3 software projects of a hospital (illustrative)	(Yeh et al., 2006)
Decision model for evaluating project implementation success	Global Efficiency Factors (GEF) approach	It can be used in each project phase to evaluate the state-of-the-art of the project. Enables the management of the complexity of ERP implementation projects more efficiently and effectively. It promotes knowledge exchange. It can be used in any kind of ERP implementation, independently of the project size or implementation strategy. It can be used as a generic model. Note: The implications of using the model in practice are discussed.	No limitations were described by the authors.	ERP projects implementation Sample: 5 SAP ERP projects (experimentation in practice)	(Bokovec et al., 2011)
Methodology and framework for project management performance measurement	Multi-criteria Decision Aiding - Constructivist (MCDA-C); MACBETH	It makes it possible to visualize the criteria that must be taken into account according to decision makers' values. It supports the ordinal and cardinal measurement of project performance. It helps in negotiations between stakeholders. Has the capacity to structure and evaluate the dimensions considered relevant by the project’s actors. Has the capacity to disseminate the knowledge generated. Enables the identification, organization, measurement, and integration of criteria judged necessary and sufficient by the decision-makers. Enables to present the degree of success of a project, graphically and numerically. Contributes to foreseeing the opportunities for project improvement. Serves as a basis for decision-making, thus improving the decision process. Note: The implications of using the model in practice are discussed.	The present research is tightly centered on the technical concerns of product development. The model needs to be adapted, even in contexts that are very similar, since it is specific to the context.	IT projects Sample: 1 software development project (experimentation in practice)	(Lacerda et al., 2011)
Project evaluation Model	Analytical Network Process (ANP)	Proposes dimensions for evaluating success, with a total of 50 criteria. ANP is utilized to determine the weights for each dimension.	No limitations were described by the authors.	Software projects Sample: 3 projects (illustrative)	(Yang et al., 2009)
Approach to measure project performance	Value Focused Thinking (VFT); Goal Question Metric (GQM)	It can be used from the project definition phase. It helps to identify and provides clarity to the objectives. Enables to determine project performance measures that are based on stakeholders' objectives regarding the project. Integrates areas of assessment of project value or stakeholders' expectations by focusing on the elicitation of objectives from the project stakeholders and the alignment of these with developed measures. Provides an analytical aid on the interrelationships between project fundamental objectives and project means objectives. May alert project stakeholders on missing or incomplete objectives and/or measures. Has a reduced reliance on predetermined criteria, which has the advantage of not confining the thinking of the stakeholders. Incorporates diverse stakeholders' perspectives to frame how the project is evaluated, instead of consideration for only the most active or most influential, for example. There is a formal alignment of perspectives of measures to the identified objectives. Note: Presents the perspective of the company’s participants regarding the advantages of the approach in practice.	A single project case was used to demonstrate the underlying principles of the approach, and it was not used from the project definition phase as it should. The research is in its formative stage and requires further evaluation and refinement. It needs in-depth project case examination in diverse environments and contexts, along with exploration on alternative strategies to help practitioners to determine the outcome of their projects. Managing conflicts among stakeholders also needs to be further explored and integrated into the approach.	IS projects Sample: 1 project (experimentation in practice)	(Barclay and Osei-Bryson, 2009)
Method for assessing project success	Subjective Evaluation Factors (SEF)	It may be used during the early phases of a project. It is based on subjective factors. As new projects are conducted, the experience base is enriched, enabling to use of historical data from prior projects to determine the classification model and estimate the project characteristic early, and predict certain success indicators. Enables to assess the relationship between project characteristics and success. Projects can be classified into classes based on project characteristics and success indicators. It is possible to indicate uncertainty in the classification. It is useful for prediction purposes. Capable of identifying key project characteristics which influence a certain success variable. It provides an opportunity to define important project characteristics and success variables in each project. It is independent of the organization and specific measures. It is useful to reduce project risk.	Further applications include the weighting and prioritization of different success indicators.	Software projects Sample: Two illustrative case studies: 12 software projects for one company; plus 46 projects from a database (illustrative)	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
Project success measurement framework	-	It is focused on measuring the success of the overall project (outputs and outcomes). Project success is seen from four dimensions: project technical, stakeholder acceptance, product quality, and organizational benefits.	It requires evaluation in several types of IT projects.	IT projects (theoretical)	(Sulistiyani and Tyas, 2019)

Table 2.

Benefits/limitations of models/methods for the evaluation of the non-IS project success.

Model/Method/Framework	Uses/Applies/Is based on	Benefits (according to the authors)	Limitations/Further developments (according to the authors)	Project Types/Sample	References
Performance-based approach to project assignment and performance evaluation	Data Envelopment Analysis (DEA)	Focuses on measuring the relative performance efficiency of the projects and the project managers. Objective-oriented preference-based assignment process. Efficiency-based evaluation process. It facilitates objective-focused management of projects. It can be easily implemented in a project-based company. It is designed for a multi-project environment. It provides a proactive and transparent mechanism for managing the assignment of new project managers and for evaluating the performance efficiency of the completed projects and their responsible project managers.	There is the risk of opportunistic behavior of some project managers. Some settings may require other measures.	- Sample: 20 completed projects and 10 new projects (illustrative)	(Xu and Yeh, 2014)
Model for determining the efficiency of projects	Data Envelopment Analysis (DEA)	Enable to determine the project efficiency score.	No limitations were described by the authors.	Construction projects Sample: 12 projects (illustrative)	(Rani et al., 2020)
Approach for evaluating projects	Step-Wise Weight Assessment Ratio Analysis (SWARA); Interval-Value Fuzzy extension of Additive Ratio Assessment (ARAS); fuzzy Delphi	It aims to find a balance between sustainable development, environmental impact, and human well-being, i.e., to find symmetry regarding goals, risks, and constraints to cope with complicated problems. It overcomes the inherent uncertainty of the projects.	In future applications, it can be used aggregate operators to integrate expert opinions and hesitant fuzzy sets to further investigate the sensitive analysis of decisions. It is assumed that criteria are independent of each other - future research can examine the existence of dependencies and use appropriate methods to lead with them.	Oil and gas well drilling projects Sample: 7 projects (illustrative)	(Dahooie et al., 2018)
Model for evaluating project success	Fuzzy Synthetic Evaluation	It allows evaluating the success levels of projects reliably. Enables to compare with more accuracy the success levels of two or more projects on the same basis.	Research with low sample size. The success criteria scores for the project are mainly derived based on secondary data.	Public-private partnership projects Sample: 1 project (illustrative)	(Osei-Kyei and Chan, 2018)
Success evaluation model for project management	Fuzzy Logic	Allows the simulation of the uncertainty that is always involved in projects. It is recommended for use primarily in the implementation phase of the project cycle and then repeatedly after each project milestone is reached. The model can be used for experimentation, providing additional information about possible project development and, in some cases, alarm signals to support future decision-making. Has the ability to record inexact (vague) concepts that project managers use in their natural language in the design and implementation of projects. It provides a tool for the measurement of selected project processes.	No limitations were described by the authors.	- Sample: 1 fictional project (illustrative)	(Doskočil et al., 2016)
Model for the evaluation of projects	Fuzzy Logic	It can be used to implement continuous quality improvement of developing projects in manufacturing Small and Medium Enterprises (SMEs). The project’s success is determined in an exact way by using fuzzy logic principles. Every solution that is obtained in an exact way is less burdened by the subjective thinking of decision-makers. It gives a chance to the management team to define measures to enhance the project’s success. It is a flexible method since it can be easily extended. It is a user-friendly methodology.	There is a need for well-structured project performance. There is a need for well-structured management teams.	Projects for business process quality improvement Sample: 60 SMEs' projects (illustrative)	(Tadić et al., 2015)
Model for evaluating project success	Principal Component Analysis (PCA); Fuzzy Neural Network (FNN)	It provides a reliable and credible way to evaluate success and enhances work efficiency. It provides a reliable method for post-evaluation.	It requires more research.	Electric power engineering projects Sample: 1 project (illustrative)	(Duan et al., 2008)
Method for evaluating the project success	Variable Weight Grey Cluster	Simple and easy to understand. It provides a reliable and credible way for success evaluation. It provides a reliable method for post-evaluation.	For applying this method in another project, it is required to adapt some parameters (index). It needs more research.	Power plant construction projects Sample: 1 project (illustrative)	(Huang et al., 2008)
Project management performance evaluation method	Back Propagation Neural Network	Integrates the main factors that can affect project management performance. Can reflect the complexity of the nonlinear relationship between the project schedule, quality, cost, safety, and project performance. It makes project management performance evaluation more objective and effective.	No limitations were described by the authors.	Construction projects Sample: 3 projects (illustrative)	(Du, 2015)
Project performance evaluation model	PROMETHEE GDSS; GAIA	Higher adherence of the project’s team due to the participation of different decision-makers, each one representing different interests inside the project. There is more dependability in analyzing judgments due to the use of preference functions. There is a geometric interpretation of data, enabling the identification of similarities and conflicts of preference between decision-makers and criteria, which can be used to check a consensus among them. Note: Discusses the practical results and implications of using the model.	The adopted scales do not use some relevant criteria and use qualitative levels (concerning some criteria), with unclear definitions, which can result in the imprecision of judgments.	Satellite projects Sample: 1 project (experimentation in practice)	(Gonçalves and Belderrain, 2012)
Model for measuring the project success	Key Performance Indicators (KPI); Analytic Hierarchy Process (AHP)	Contributes to reflection and learning, given the indicators analysis. The measurement approach developed for the model serves to generate new measurement models. Such models may incorporate other themes, contextual factors, and considerations different from those performed in the case. Assist investors in decision-making when planning to implement a photovoltaic project by measuring the achievement of their project’s objectives, which in turn is the project’s success rate. The determining criteria are identified through a weighting system. There is the possibility for reflection and learning by decision-makers. Enables to compare the success of different projects and discuss the factors that cause failure. It has a dependent variable conception (global index of successful investment in photovoltaic systems) that allows explanatory relationships to be established with external factors to the proposed measurement model.	It is suggested, in future applications, other performance measurements, and multiple-criteria decision analysis weighting methods to make the diagnosis even more assertive.	Distributed small-scale photovoltaic systems projects Sample: 32 projects (illustrative)	(Rigo et al., 2020)
Approach for evaluating project success	M-TOPSIS	Allows data that has been relativized and transposed according to the planned parameters to be directly characterized as a percentage of its general success.	The results from this method are very sensitive to incorrectly input data.	Construction projects Sample: 1 fictional project (illustrative)	(Pinter and Pšunder, 2013)
Project Balance Evaluation Method (PBE) for project performance evaluation	Balanced Score Card (BSC); Earned Value Management (EVM)	It is suitable for project monitoring. It can be used for communication and feedback among the project’s stakeholders. Can introduce operation’s weak spots and the place of spending extra resources. Risk evaluation can be applied in a better way than the traditional BSC and EVM approaches. It shows that risk management should be executed throughout the whole project and all the important intervals of the project. It is easy to understand and categorize problems in a project’s life cycle. It is designed to persuade managers to follow the occurrence of problems and their reasons. Considers time, cost, and satisfaction in all phases of the project’s lifecycle.	No limitations were described by the authors.	- (theoretical)	(Sharifi and Nik, 2016)
Template for measuring project success	Delphi; Priority Evaluation Method	It is a useful tool for measuring success. Note: The obtained results in practice are discussed. It will be applied in further projects in the same organization.	No limitations were described by the authors.	Water reservoir projects Sample: 5 projects (experimentation in practice)	(Ismail, 2020)
Project performance assessment method	Activity Based Cost (ABC)	It provides accurate information. It helps to optimize the decision of public expenditure.	No limitations were described by the authors.	Public projects (theoretical)	(Hong-xiong et al., 2010)
Project performance evaluation approach	MACBETH	Enable to identify the most important criteria. Models the elements of the project into a supervisory level group to facilitate decision-making by the project manager. Measurements produce levels of monitoring classification of project element groups.	Future applications can consider the comparison of performance between the two joint performances between elements so that the comparison between the combined performance ratings can be more accurate. The approach was not tested in an ongoing project. The approach does not consider elements such as stakeholders or project sponsorship in the performance evaluation.	Drilling projects in oil and gas Sample: 1 project (illustrative)	(Dachyar and Pratama, 2014)
Project evaluation tool	MACBETH	The evaluation problem can be put in terms of a multi-dimensional model describing performances without forcing conversion to monetary units. Social phenomena can be analyzed quantitatively through a value model and fully described qualitatively by means of constructed descriptors. Compels donors to resolve tradeoffs between evaluation criteria attached to equally important strategic concerns. It constitutes a powerful communication tool amongst stakeholders. Avoid possible misunderstandings that could arise from the ambiguous use of terms. There is a didactical capability of the tool. It improves the understanding of the operational implications of cross-cutting issues in day-to-day operations. Note: Presents and discusses the results of the application of the tool in practice.	No limitations were described by the authors.	Rural development programs Sample: 1 program (experimentation in practice)	(Dachyar and Pratama, 2014; Sanchez-Lopez et al., 2012)
Process for risk-informed performance assessment of projects	Fuzzy numbers	It is expected to provide practitioners with a relatively easy approach for integrating both risk and performance management. Incorporates risk management. It integrates the benchmarking approach through indicators with a robust yet simple mathematical structure. The adoption of fuzzy numbers for the modeling of risk provides a sound way of dealing with the incompleteness of data when this occurs. It performs risk-informed performance control and future performance estimations. Provides quantified outputs that can be used by the construction project’s management team for deciding appropriate interventions with respect to the achieved and desired project’s performance. It is comprehensive and easy to apply the methodology. Allows the revision of construction projects' performance goals based on the success of managing risks in these projects.	To achieve better modeling of the performance variables should be included in their interdependencies, as well as a better calibration of the several weighting factors addressed in the methodology shall advance even more risk-informed performance assessment of construction projects.	Construction projects Sample: 1 fictional case (illustrative)	(Papanikolaou and Xenidis, 2020)
Model for multi-criteria analysis of project performance	-	It is easy to compare projects, not only at the end of the project but also during the project life cycle. The information received is useful for strategic planning, quality management, solving the tasks of resource allocation, and motivational project evaluation. There is the possibility of aggregating both quantitative and qualitative criteria in the evaluation process. There is the possibility of analyzing complex problems. There is the possibility to make clear evidence of decisions. There is the possibility for decision-makers to participate actively in the decision-making process.	No limitations were described by the authors.	Construction projects Sample: 7 projects (illustrative)	(Zavadskas et al., 2014)
System-of-Systems Framework for project performance assessment	-	Facilitates considering dynamic behaviors, uncertainty, and interdependence between constituents in engineering projects, by employing two fundamental principles: base-level abstraction and multi-level abstraction. It provides a formalized approach to base-level entities, their attributes, and interactions in complex project systems. Based on formalized abstraction, different modeling and analytical tools can be better implemented in order to study various phenomena affecting project performance. Projects can be better planned in complex and uncertain environments. There is a better forecast and control of project performance by monitoring the dynamic interdependences and interactions in project systems. Enables the understanding of complex phenomena through conducting a bottom-up analysis.	The implementation in large complex projects requires abstracting many base-level entities, as well as their attributes and interdependencies. The computational complexity increases with the increase in the number of base-level entities and attributes abstracted and modeled.	Engineering projects Sample: 1 project (illustrative)	(Zhu and Mostafavi, 2018)
Framework for Project Evaluation on Strategic, Tactical and Operational Levels (PESTOL)	-	The model combines elements from existing evaluation models to form an improved framework. The evaluation addresses the operational, tactical, and strategic levels. It covers the whole project life cycle. It is suitable for the ex-post evaluation of projects.	There may be partiality in the judgments. There is subjectivity in the scores that reflect stakeholders' perceptions. Further research is needed to develop a systematic method based on the model in order to reflect the evaluation measures and their rationality. There is also needed further research on how to link the ex-post evaluation model to ex-ante, monitoring, mid-term, and terminal evaluations.	Highway construction megaprojects Sample: 1 project (illustrative)	(Zidane et al., 2016)

It is worth stressing that the identified models/methods are presented by the original authors as “models”, “frameworks”, “approaches”, “methods”, “methodologies”, “tools”, and “techniques”, in general using these terms indistinctively.

To note that the application area (IS or non-IS) does not mean that a model/method is restricted to IS or non-IS projects evaluation. Conversely, it means that the application area used to describe the model/method is (or is not) an IS project. For instance, the models based on DEA (Wray and Mathieu, 2008; Xu and Yeh, 2014) or MACBETH (Lacerda et al., 2011; Dachyar and Pratama, 2014; Sanchez-Lopez et al., 2012) can be used in virtually all project areas/types. However, the conclusion of whether a specific model/method that was applied to IS projects is suitable for non-IS projects evaluation (or vice-versa) requires further research. For more details on each model/method, it is recommended to read the original sources (some references present processes or examples for using the model/method in practice).

5. Discussion

Based on the analysis of the identified models/methods, this section discusses the underlying techniques, the empirical basis, and the benefits/limitations of the models/methods as described by the authors. In the end, significant insights and future directions for research are discussed.

5.1. Models/methods' underlying techniques

Table 3 lists a summary of the models/methods' underlying techniques. For better reference, the techniques were grouped by purpose/main characteristics. Some techniques are repeated since they fall into more than one group.

Table 3.

Complete list of techniques underlying the models/methods for evaluating the success of projects.

Group	Technique	Project Type	References
Multi-Criteria Decision Analysis	Analytic Hierarchy Process (AHP)	Photovoltaic systems projects	(Rigo et al., 2020)
	Analytical Network Process (ANP)	IT projects	(Yang et al., 2009)
	Fuzzy Analytical Network Process (FANP)	IT projects	(Hermawan et al., 2016)
	GAIA	Satellite projects	(Gonçalves and Belderrain, 2012)
	Global Efficiency Factors (GEF)	IT projects	(Bokovec et al., 2011)
	Interval-Value Fuzzy extension of Additive Ratio Assessment (ARAS)	Oil and gas projects	(Dahooie et al., 2018)
	MACBETH	IT projects Oil and gas projects Rural development programs	(Lacerda et al., 2011; Dachyar and Pratama, 2014; Sanchez-Lopez et al., 2012)
	Modified Technique for Order of Preference by Similarity to Ideal Solution (M-TOPSIS)	Construction projects	(Pinter and Pšunder, 2013)
	PROMETHEE	Satellite projects	(Gonçalves and Belderrain, 2012)
	Value Focused Thinking (VFT)	IT projects	(Barclay and Osei-Bryson, 2009)
Fuzzy Logic/fuzzy numbers	Fuzzy Analytical Network Processing (FANP)	IT projects	(Hermawan et al., 2016)
	Fuzzy Delphi	Oil and gas projects	(Dahooie et al., 2018)
	Fuzzy Logic	Projects in general Projects for business processes improvement	(Doskočil et al., 2016; Tadić et al., 2015)
	Fuzzy Logic Expert approach, based on Fuzzy Linguistic Parameters and Fuzzy values	IT projects	(Pujari and Seetharam, 2015)
	Fuzzy Neural Network (FNN)	Electric power engineering projects	(Duan et al., 2008)
	Fuzzy numbers	Construction projects	(Papanikolaou and Xenidis, 2020)
	Fuzzy Synthetic Evaluation	Public-private partnership projects	(Osei-Kyei and Chan, 2018)
	Hesitant Fuzzy Set (HFS)	IT projects	(Basar, 2020)
	Interval-Value Fuzzy extension of Additive Ratio Assessment (ARAS)	Oil and gas projects	(Dahooie et al., 2018)
	Ordered Weighted Averaging (OWA)	IT projects	(Yeh et al., 2006)
	Variable Weight Grey Cluster	Construction projects	(Huang et al., 2008)
Performance management/measurement	Activity Based Cost (ABC)	Public projects	(Hong-xiong et al., 2010)
	Balanced Scorecard (BSC)	IT projects Projects in general	(Basar, 2020; Hermawan et al., 2016; Sharifi and Nik, 2016)
	Earned Value Management (EVM)	Projects in general	(Sharifi and Nik, 2016)
	Goal Question Metric (GQM)	IT projects	(Barclay and Osei-Bryson, 2009)
	Key Performance Indicators (KPI)	Photovoltaic systems projects	(Rigo et al., 2020)
Neural Networks	Back Propagation Deep Belief Network (BP-DBN)	IT projects	(Nguvulu et al., 2011)
	Back Propagation Neural Network (BPNN)	Construction projects	(Du, 2015)
	Deep Belief Networks (DBN)	IT projects	(Nguvulu et al., 2012)
	Fuzzy Neural Network (FNN)	Electric power engineering projects	(Duan et al., 2008)
Criteria prioritization and weighting	Delphi	Water reservoir projects	(Ismail, 2020)
	Fuzzy Delphi	Oil and gas projects	(Dahooie et al., 2018)
	Priority Evaluation Method (PEM)	Water reservoir projects	(Ismail, 2020)
	Step-Wise Weight Assessment Ratio Analysis (SWARA)	Oil and gas projects	(Dahooie et al., 2018)
Efficiency measurement	Data Envelopment Analysis (DEA)	IT projects Construction projects Projects in general	(Wray and Mathieu, 2008; Xu and Yeh, 2014; Rani et al., 2020)
Other	Principal Component Analysis (PCA)	IT projects Electric power engineering projects	(Yeh et al., 2006; Duan et al., 2008)
	Subjective Evaluation Factors (SEF)	IT projects	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
	Miscellaneous/not specified	IT projects Construction projects Engineering projects	(Sulistiyani and Tyas, 2019; Zavadskas et al., 2014; Zhu and Mostafavi, 2018; Zidane et al., 2016)

In Table 3 the prevalence of techniques of the group multi-criteria decision analysis is easily noticeable (e.g., Rigo et al. (2020)) as well as the application of fuzzy logic (e.g., Basar (2020)). This is understandable since the evaluation of the success of projects is multi-dimensional (Silvius and Schipper, 2015), involving quantitative and qualitative criteria and subjective human judgments (Cuellar, 2010).

This synthesis of techniques presents three main contributions: on the one hand, it identifies alternative ways for the practical evaluation of the success of projects by using techniques that have already been explored; it presents opportunities for research replication (in the same or new project types where the techniques have been applied); it also helps to identify gaps in research and opportunities to focus on techniques not yet explored or applied.

5.2. Empirical grounding

Even though it is out of the scope of this study to carry out independent experimentation of each model/method, it is important to reflect on the empirical grounding of the models/methods identified.

As previously noted, by looking at literature, project evaluation in practice seems to be an underrated process since most projects are evaluated with a limited set of criteria (typically related to time, scope, and budget) – and many times not formally. Furthermore, the used models/methods for the evaluation or the procedures for measuring success are not fully documented. Even so, the project’s success or failure is reported (Standish, 2018; Marnewick, 2012; Varajão and Carvalho, 2018; Pereira et al., 2022). On the other hand, our study shows a rich set of models/methods that aim to support the evaluation of success following more or less sophisticated approaches.

However, results also show that many techniques extant in the literature are, in their essence, mostly theoretical exercises without evidence of extensive use in practice. This is evident when analyzing the column “project types/sample” in Tables 1 and 2. Only a few cases (namely, Basar (2020); Bokovec et al. (2011); Lacerda et al. (2011); Barclay and Osei-Bryson (2009); Gonçalves and Belderrain (2012); Ismail (2020); Dachyar and Pratama (2014); Sanchez-Lopez et al. (2012)) describe the use of the proposed model/method in real projects and discuss the practical implications of its use. In other words, only about 20% of the published models/methods for evaluating project success present empirical evidence of its feasibility and real usefulness. It is necessary to remark that, even in these cases, overall, only one project or a small set of projects of the same entity are reported, not being possible to ascertain if the model/method has been used beyond the reported project(s) (one exception is Ismail (2020)).

The lack of pieces of evidence of the practical use of some models/methods can be explained, in some cases, by their novelty. In other cases, the models/methods may not be used in practice due to the difficulty of applying them or even due to not being known. Some evaluation models/methods are, as expected, more complex than others, and it is fair to mention that some of them are well structured and clearly explained in the original source. Contrarily, unfortunately, in most cases, the articles report only the very early stages of the development of the models/methods, and the descriptions are quite incomplete. They do omit important details, such as the steps that should be followed to perform the evaluation. These are not here identified here due to academic courtesy, but at least more than half of the identified references present deficient academic qualities, which can jeopardize their adoption and application in practice (some were included in this review only because the underlying idea is relevant).

The literature review reveals an evident lack of ongoing project case studies and replication studies. Consequently, it is recommended that more research is carried out, presenting case studies well-grounded in empirical data. There is also the need for new surveys (e.g., questionnaire-based) to create a real picture of organizations' practices (models/methods) concerning models/methods used for project success evaluation.

5.3. Benefits and limitations of the surveyed models

Table 4 summarizes the benefits of the models/methods highlighted in the reviewed articles, ordered by frequency. Some of the most mentioned benefits by authors are: decision-making support; ability to compare projects; ability to specify metrics; multi-dimensional evaluation support; reliability and accuracy; clarity of the objectives; risk management support; incorporation of stakeholders' perspectives; allowance of simulation and forecast; project monitoring support; ability to assess criteria weights; inclusion of subjective measures; inclusion of objective measurements; facilitation of communication; contribution to reflection and learning; among others. These are strong reasons for practitioners to take it into account in their projects.

Table 4.

Reported benefits of the models/methods for evaluating the success of projects.

Group	Benefits	References
Supports decision-making	Contributes to foreseeing the opportunities for project improvement.	(Lacerda et al., 2011)
	Serve as a basis for decision-making, thus improving the decision process.	(Lacerda et al., 2011)
	Capable of identifying key project characteristics which influence a certain success variable.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
	Helps to optimize the decision of public expenditure.	(Hong-xiong et al., 2010)
	Facilitates objective-focused management of projects.	(Xu and Yeh, 2014)
	There is a possibility of analyzing complex problems.	(Zavadskas et al., 2014)
	There is a possibility to make clear evidence of decisions.	(Zavadskas et al., 2014)
	There is a possibility for decision-makers to participate actively in the decision-making process.	(Zavadskas et al., 2014)
	Facilitates considering dynamic behaviors, uncertainty, and interdependence between constituents in engineering projects, by employing two fundamental principles: base-level abstraction and multi-level abstraction.	(Zhu and Mostafavi, 2018)
	Assist investors in decision-making when planning to implement a photovoltaic project by measuring the achievement of their project’s objectives, which in turn is the project’s success rate.	(Rigo et al., 2020)
	Projects can be better planned in complex and uncertain environments.	(Zhu and Mostafavi, 2018)
	The information received is useful for strategic planning, quality management, solving the tasks of resource allocation, and motivational project evaluation.	(Zavadskas et al., 2014)
	Provides quantified outputs that can be used by the construction project’s management team for deciding appropriate interventions with respect to the achieved and desired project performance.	(Papanikolaou and Xenidis, 2020)
	Models the elements of the project into a supervisory level group to facilitate decision-making by the project manager.	(Dachyar and Pratama, 2014)
	Enables to assess the relationship between project characteristics and success.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
Enables to compare projects	Projects are directly compared.	(Wray and Mathieu, 2008)
	Enables to link an inefficient project to its benchmark efficient project(s).	(Wray and Mathieu, 2008)
	Enables ordering projects by efficiency.	(Wray and Mathieu, 2008)
	Enables to evaluate the relative performance of projects.	(Wray and Mathieu, 2008)
	Projects can be classified into classes based on project characteristics and success indicators.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
	Enables to compare with more accuracy the success levels of two or more projects on the same basis.	(Osei-Kyei and Chan, 2018)
	Focuses on measuring the relative performance efficiency of the projects and of the project managers.	(Xu and Yeh, 2014)
	It is designed for a multi-project environment.	(Xu and Yeh, 2014)
	It is easy to compare projects, not only at the end of the project but also during all the project life cycle.	(Zavadskas et al., 2014)
	Enables to compare the success of different projects and discuss the factors that cause failure.	(Rigo et al., 2020)
	Integrates the benchmarking approach through indicators with a robust yet simple mathematical structure.	(Papanikolaou and Xenidis, 2020)
Enables to specify metrics	Enables to specify performance metrics.	(Basar, 2020)
	It allows visualizing the criteria that must be taken into account according to decision makers' values.	(Lacerda et al., 2011)
	Has a reduced reliance on predetermined criteria, which has the advantage of not confining the thinking of the stakeholders.	(Barclay and Osei-Bryson, 2009)
	Enables the identification, organization, measurement, and integration of criteria judged necessary and sufficient by the decision-makers.	(Lacerda et al., 2011)
	Provides an opportunity to define important project characteristics and success variables in each project.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
	It gives a chance to the management team to define measures to enhance the project’s success.	(Tadić et al., 2015)
	Enables to identify the most important criteria.	(Dachyar and Pratama, 2014)
	The number of criteria is reduced by PCA but preserves the original information.	(Yeh et al., 2006)
	It is independent of the organization and specific measures.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
Measures the success of a project	Measures the success of a project.	(Pujari and Seetharam, 2015)
	Produces an efficiency score for projects.	(Wray and Mathieu, 2008)
	Enables to present the degree of success of a project, graphically and numerically.	(Lacerda et al., 2011)
	It is focused on measuring the success of the overall project (outputs and outcomes).	(Sulistiyani and Tyas, 2019)
	Enables to determine the project efficiency score.	(Rani et al., 2020)
	Allows to evaluate the success levels of projects reliably.	(Osei-Kyei and Chan, 2018)
	Provides a tool for measurement of selected project processes.	(Doskočil et al., 2016)
	Allows data that has been relativized and transposed according to the planned parameters to be directly characterized as a percentage of its general success.	(Pinter and Pšunder, 2013)
	Has a dependent variable conception (global index of successful investment in photovoltaic systems) that allows explanatory relationships to be established with external factors to the proposed measurement model.	(Rigo et al., 2020)
Supports a multi-dimensional evaluation	Handles multiple project inputs.	(Wray and Mathieu, 2008)
	Handles multiple project outputs.	(Wray and Mathieu, 2008)
	Measures performance from a multi-domain perspective.	(Hermawan et al., 2016)
	Provides an analytical aid on the interrelationships between project fundamental objectives and project means objectives.	(Barclay and Osei-Bryson, 2009)
	Has the capacity to structure and evaluate the dimensions considered relevant by the project’s actors.	(Lacerda et al., 2011)
	Project success is seen from four dimensions: project technical, stakeholder acceptance, product quality, organizational benefits.	(Sulistiyani and Tyas, 2019)
	Proposes dimensions for evaluating the success.	(Yang et al., 2009)
	The evaluation problem can be put in terms of a multi-dimensional model describing performances without forcing conversion to monetary units.	(Dachyar and Pratama, 2014; Sanchez-Lopez et al., 2012)
It can be used during the project lifecycle	It can be used from the project definition phase.	(Barclay and Osei-Bryson, 2009)
	It may be used during the early phases of a project.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
	Enables to compute reliable monthly project performance indicators based on the basic project elements data.	(Nguvulu et al., 2011)
	It covers the whole project life cycle.	(Zidane et al., 2016)
	It can be used in each project phase to evaluate the project’s state-of-the-art.	(Bokovec et al., 2011)
	It can be used to implement continuous quality improvement in developing projects.	(Tadić et al., 2015)
	It is recommended for use primarily in the implementation phase of the project cycle and then repeatedly after each project milestone is reached.	(Doskočil et al., 2016)
	Considers time, cost, and satisfaction in all phases of the project’s lifecycle.	(Sharifi and Nik, 2016)
Reliable and accurate	Provides a reliable and credible way to evaluate success and enhances work efficiency.	(Duan et al., 2008)
	Provides a reliable and credible way to evaluation of success.	(Huang et al., 2008)
	The project’s success is determined in an exact way by using fuzzy logic principles.	(Tadić et al., 2015)
	Provides accurate information.	(Hong-xiong et al., 2010)
	It provides a proactive and transparent mechanism for managing the assignment of new project managers and for evaluating the performance efficiency of the completed projects and their responsible project managers.	(Xu and Yeh, 2014)
	Every solution that is obtained in an exact way is less burdened by the subjective thinking of decision-makers.	(Tadić et al., 2015)
	Provides a formalized approach to base-level entities, their attributes, and interactions in complex project systems.	(Zhu and Mostafavi, 2018)
	Supports the ordinal and cardinal measurement of project performance.	(Lacerda et al., 2011)
It helps to identify and provides clarity to the objectives	It helps to identify and provides clarity to the objectives.	(Barclay and Osei-Bryson, 2009)
	May alert project stakeholders on missing or incomplete objectives and/or measures.	(Barclay and Osei-Bryson, 2009)
	Increases the objectivity of the project evaluation process.	(Nguvulu et al., 2011)
	Provides a more rational and objective evaluation approach.	(Nguvulu et al., 2011)
	Enables objective-oriented preference-based assignment process.	(Xu and Yeh, 2014)
	Avoid possible misunderstandings that could arise from the ambiguous use of terms.	(Dachyar and Pratama, 2014; Sanchez-Lopez et al., 2012)
Supports risk management	It is expected to provide practitioners with a relatively easy approach for integrating both the risk and performance management.	(Papanikolaou and Xenidis, 2020)
	It shows that risk management should be executed throughout the whole project and all the important intervals of the project.	(Sharifi and Nik, 2016)
	Incorporates risk management.	(Papanikolaou and Xenidis, 2020)
	It is useful to reduce project risk.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
	Risk evaluation can be applied in a better way than the traditional BSC and EVM approaches.	(Sharifi and Nik, 2016)
	Allows the revision of construction projects' performance goals based on the success of managing risks in these projects.	(Papanikolaou and Xenidis, 2020)
Easy to understand and simple to use	Simple and easy to understand.	(Huang et al., 2008)
	Reduces the complexity of software project evaluation.	(Yeh et al., 2006)
	It is a user-friendly methodology.	(Tadić et al., 2015)
	It can be easily implemented in a project-based company.	(Xu and Yeh, 2014)
	It is easy to understand and categorize problems in a project’s life cycle.	(Sharifi and Nik, 2016)
	It is comprehensive and easy to apply the methodology.	(Papanikolaou and Xenidis, 2020)
Incorporates stakeholders perspectives	Enables to determine project performance measures that are based on stakeholders' objectives.	(Barclay and Osei-Bryson, 2009)
	Incorporates diverse stakeholders' perspectives to frame how the project is evaluated.	(Barclay and Osei-Bryson, 2009)
	There is a formal alignment of perspectives of measures to the identified objectives.	(Barclay and Osei-Bryson, 2009)
	Helps in negotiations between stakeholders.	(Lacerda et al., 2011)
	Higher adherence of the project’s team due to the participation of different decision-makers, each one representing different interests inside the project.	(Gonçalves and Belderrain, 2012)
	Geometric interpretation of data, enabling the identification of similarities and conflicts of preference between decision-makers and criteria, which can be used to check a consensus among them.	(Gonçalves and Belderrain, 2012)
Allows simulation and forecast	Allows the simulation of the uncertainty that is always involved in projects.	(Doskočil et al., 2016)
	The model can be used for experimentation, providing additional information about possible project development and, in some cases, alarm signals to support future decision-making.	(Doskočil et al., 2016)
	It is useful for prediction purposes.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
	Performs risk-informed performance control and future performance estimations.	(Papanikolaou and Xenidis, 2020)
	Better forecast and control of project performance by monitoring the dynamic interdependences and interactions in project systems.	(Zhu and Mostafavi, 2018)
Supports project monitoring	It is suitable for project monitoring.	(Sharifi and Nik, 2016)
	Can introduce operation’s weak spots and the place of spending extra resources.	(Sharifi and Nik, 2016)
	It is designed to persuade managers to follow the occurrence of problems and their reasons.	(Sharifi and Nik, 2016)
	Improves the understanding of the operational implications of cross-cutting issues in the day-to-day operations.	(Dachyar and Pratama, 2014; Sanchez-Lopez et al., 2012)
	Measurements produce levels of monitoring classification of project element groups.	(Dachyar and Pratama, 2014)
Enables assessing criteria weights	Enables to find criteria weights to measure the performance in fuzzy environments.	(Basar, 2020)
	Enables to determine weighting priorities for key performance indicators.	(Hermawan et al., 2016)
	It provides flexibility for deciding the weight distribution among criteria.	(Yeh et al., 2006)
	The determining criteria are identified through a weighting system.	(Rigo et al., 2020)
	The weights of resultant criteria are obtained by employing OWA.	(Yeh et al., 2006)
Comprises subjective measures	Enables subjective weighting of project performance factors.	(Basar, 2020)
	Comprises qualitative and quantitative measurements.	(Hermawan et al., 2016)
	It is based on subjective factors.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
	There is the possibility of aggregating both quantitative and qualitative criteria in the evaluation process.	(Zavadskas et al., 2014)
Comprises objective measurements	Comprises qualitative and quantitative measurements.	(Hermawan et al., 2016)
	It makes project management performance evaluation more objective and effective.	(Du, 2015)
	Social phenomena can be analyzed quantitatively through a value model and fully described qualitatively by means of constructed descriptors.	(Dachyar and Pratama, 2014; Sanchez-Lopez et al., 2012)
	There is the possibility of aggregating both quantitative and qualitative criteria in the evaluation process.	(Zavadskas et al., 2014)
Facilitates communication	Has the capacity to disseminate the knowledge generated.	(Lacerda et al., 2011)
	Promotes knowledge exchange.	(Bokovec et al., 2011)
	Constitutes a powerful communication tool amongst stakeholders.	(Dachyar and Pratama, 2014; Sanchez-Lopez et al., 2012)
	It can be used for communication and feedback among the project’s stakeholders.	(Sharifi and Nik, 2016)
Automatically assigns weights to project elements	Automatically assigns weights and allocates ceiling scores to the project elements, capturing interactions between the project elements.	(Nguvulu et al., 2012)
	It does not depend on humans to manually assign weights.	(Nguvulu et al., 2012)
	Eliminates the manual assignment of model weights.	(Nguvulu et al., 2011)
	Automatically assigns weights to the project elements during model training, with the possibility of self-adjustment as new data is presented to the operational model.	(Nguvulu et al., 2011)
Contributes to reflection and learning	Contributes to reflection and learning, given the indicators analysis.	(Rigo et al., 2020)
	It is a didactic tool.	(Dachyar and Pratama, 2014; Sanchez-Lopez et al., 2012)
	There is the possibility for reflection and learning by decision-makers.	(Rigo et al., 2020)
Deals with uncertainty	It is possible to indicate uncertainty in the classification.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
	Has the ability to record inexact (vague) concepts that project managers use in their natural language in the design and implementation of projects.	(Doskočil et al., 2016)
	Overcomes the inherent uncertainty of the projects.	(Dahooie et al., 2018)
Has the ability to store learned knowledge	Has the ability to store learned knowledge.	(Nguvulu et al., 2012)
	As new projects are conducted, the experience base is enriched, enabling to use of historical data from prior projects to determine the classification model and estimate the project characteristic early and predict certain success indicators.	(Wohlin et al., 2000; Wohlin and Andrews, 2001)
It can be used in ex-post project evaluation	It is suitable for the ex-post evaluation of projects.	(Zidane et al., 2016) (Huang et al., 2008)
	Provides a reliable method for post-evaluation.	(Duan et al., 2008)
Flexible method	It is a flexible method since it can be easily extended.	(Tadić et al., 2015)
	It can be used as a generic model, independently of the project size or implementation strategy.	(Bokovec et al., 2011)
	The measurement approach developed for the model serves to generate new measurement models.	(Rigo et al., 2020)
Enables impartial evaluation of projects	It can be used for an impartial evaluation of projects.	(Pujari and Seetharam, 2015)
	It eliminates much of the subjectivity inherent to weighting the criteria manually.	(Nguvulu et al., 2012)
Supports complexity	Can reflect the complexity of the nonlinear relationship between the project schedule, quality, cost, safety, and project performance.	(Du, 2015)
	Enables the understanding of complex phenomena through conducting bottom-up analysis.	(Zhu and Mostafavi, 2018)
Automatically adapts to new knowledge	Has the capability to automatically adapt to new knowledge even after the model becomes fully operational.	(Nguvulu et al., 2012)
Addresses the operational, tactical, and strategic levels	The evaluation addresses the operational, tactical, and strategic levels.	(Zidane et al., 2016)
Supports incomplete data	The adoption of fuzzy numbers for the modeling of risk provides a sound way of dealing with the incompleteness of data when this is occurring.	(Papanikolaou and Xenidis, 2020)
Other	It aims to find a balance between sustainable development, environmental impact, and human well-being, i.e., to find symmetry regarding goals, risks, and constraints to cope with complicated problems.	(Dahooie et al., 2018)
	Compels donors to resolve tradeoffs between evaluation criteria attached to equally important strategic concerns.	(Dachyar and Pratama, 2014; Sanchez-Lopez et al., 2012)
	Enables to manage the complexity of ERP implementation projects more efficiently and effectively.	(Bokovec et al., 2011)
	Based on formalized abstraction, different modeling and analytical tools can be better implemented in order to study various phenomena affection project performance.	(Zhu and Mostafavi, 2018)
	Integrates the main factors that can affect the project management performance.	(Du, 2015)
	More dependability to analyze judgments due to the use of preference functions.	(Gonçalves and Belderrain, 2012)
	The model combines elements from existing evaluation models to form an improved framework.	(Zidane et al., 2016)
	Such models may incorporate other themes, contextual factors, and considerations different from those performed in the case.	(Rigo et al., 2020)
	Success factors are determined based on expert judgments.	(Basar, 2020)
	Efficiency-based evaluation process.	(Xu and Yeh, 2014)
	It is a useful tool for measuring success.	(Ismail, 2020)

Table 5 summarizes the limitations of the models/methods reported in the articles reviewed. The most mentioned limitations are: it requires further research; risk of imprecision/lack of accuracy; and limited experimentation, sample, and data. This calls for further research since most of the proposed models/methods do not have an empirical evaluation supporting what is claimed by the authors.

Table 5.

Reported limitations of the models/methods for evaluating the success of projects.

Group	Limitations	References
Requires further research	Requires further research.	(Wray and Mathieu, 2008) (Sulistiyani and Tyas, 2019) (Duan et al., 2008) (Huang et al., 2008)
	Requires further research regarding weighting and prioritization between success indicators.	(Wohlin et al., 2000; Wohlin and Andrews, 2001) (Barclay and Osei-Bryson, 2009) (Papanikolaou and Xenidis, 2020)
	Future applications may include an awarding system.	(Basar, 2020)
	Requires further research regarding the sensitive analysis of decisions.	(Dahooie et al., 2018)
	Managing conflicts among stakeholders needs to be further explored and integrated into the approach.	(Barclay and Osei-Bryson, 2009)
	Further research is needed to develop a systematic method based on the model in order to reflect the evaluation measures and their rationality.	(Zidane et al., 2016)
Risk of imprecision/lack of accuracy	There is the subjectivity of the scores that reflect stakeholders' perceptions.	(Zidane et al., 2016)
	The results are very sensitive to input data (e.g., incorrect data).	(Pinter and Pšunder, 2013) (Wray and Mathieu, 2008)
	The adopted scales do not use some relevant criteria and use qualitative levels (concerning some criteria) with unclear definitions, which can result in imprecise judgments.	(Gonçalves and Belderrain, 2012)
	Future applications can consider the comparison of performance between the two joint performance between elements so that the comparison between the combined performance ratings can be more accurate.	(Dachyar and Pratama, 2014)
	Future applications should include other performance measurements and multiple-criteria decision analysis weighting methods in order to make the diagnosis more assertive.	(Rigo et al., 2020)
	The implementation in large complex projects requires abstracting many base-level entities, as well as their attributes and interdependencies.	(Zhu and Mostafavi, 2018)
	There is the risk of opportunistic behavior of some project managers.	(Xu and Yeh, 2014)
	The approach does not consider elements such as stakeholders or project sponsorship in the performance evaluation.	(Dachyar and Pratama, 2014)
	Do not consider dependencies between criteria.	(Dahooie et al., 2018) (Papanikolaou and Xenidis, 2020)
Limited experimentation, sample, and data	It was not tested in an ongoing project.	(Dachyar and Pratama, 2014)
	Small sample size.	(Osei-Kyei and Chan, 2018)
	The success criteria scores for the project are mainly derived based on secondary data.	(Osei-Kyei and Chan, 2018)
Other	The model needs to be adapted (e.g., regarding measures) since it is specific to the context.	(Lacerda et al., 2011) (Xu and Yeh, 2014) (Huang et al., 2008)
	It can be computationally intensive.	(Wray and Mathieu, 2008) (Zhu and Mostafavi, 2018)
	There is a need for well-structured project performances and management teams.	(Tadić et al., 2015)
	Do not allow comparison to a theoretical maximum.	(Wray and Mathieu, 2008)
	Do not link the ex-post evaluation model to ex-ante, monitoring, mid-term, and terminal evaluations.	(Zidane et al., 2016)

Understanding these benefits is important to practitioners so they can be aware of the importance of adopting well-defined techniques to assess success in their projects. Limitations are also important since they may entail risks for the project evaluation. Researchers can look at both benefits and limitations to guide research: By developing and proposing new models/methods, they should try to achieve the reported benefits; On the other hand, they should explore the limitations aiming at identifying further needs for research and carrying out new research to solve the identified issues.

We highlight that nearly 45% of the considered references – oddly – do not report any limitations. Nevertheless, it should be noted that, although not mentioned by the authors, a clear limitation of several of the proposed models/methods is the fact that they have not been tested in real projects as already mentioned. Thus, their practical effects have not been studied in organizations, making it difficult to assess their real value.

These results are also useful for improving existing methods. For example, some methods have limitations that have already been solved in others. By analyzing the methods that have addressed them, it is possible to look for solutions to evolve a specific method. For example, Barclay and Osei-Bryson (2009) state that managing conflicts among stakeholders need to be further explored and integrated into their approach. On the other hand, one advantage of Lacerda et al. (2011)’s model is helping negotiations between stakeholders. Thus, looking at the advantages and limitations between models/methods, a synergy is possible that enables their mutual evolution.

5.4. Main insights and future directions for research

Although some models/methods are based on a single evaluation criterion, others use multiple criteria and more complex assessments. Some methods, such as the ones focused on the ex-post evaluation, examine the organizational performance of the project and the informational and transformational effects that result from it (Gollner and Baumane-Vitolina, 2016), but this does not happen in most cases. In practice, model/methods for success evaluation should be defined considering not only the performance during the project but also the impacts post-project (Varajão et al., 2022a). This evaluation is fundamental for confirming the achievement of the expected benefits (Slevin and Pinto, 1987).

Some models/methods are exclusively designed to be applied in IS projects because the required inputs only exist in this kind of project. In most of them, the model has been tested in a specific area but may be extendable to other areas. However, there is rare evidence of models/methods applied in more than one area or project type. This also opens avenues for replication studies.

It is noteworthy that nearly 40% of the identified models/methods were developed/applied in IS projects, possibly indicating more concern regarding measuring success in this kind of project (eventually due to the perceived low levels of success) or the need for different models/methods due to a higher diversity of projects. This is something that also requires further research.

A major limitation of many of the reported models/methods is the fact that they have not been tested in real-world projects. This demands more empirically based research, including replication and case studies.

Due to the complexity of projects and respective environments, it can be challenging to establish the models/methods most appropriate for each case. Furthermore, a model/method defined for evaluating the success of a specific project can be a customized combination of extant models/methods features.

In conclusion, as depicted in Figure 3, the selection/definition of a model/method for evaluating the success of a specific project should take into account: the specific project and project’s environment characteristics (e.g., deliverables attributes and stakeholders' reporting requirements); the success evaluation requirements (e.g., multi-criteria evaluation) and purpose (e.g., reporting success to top management), considering the expected benefits and acceptable limitations; the models/methods for project evaluation characteristics (e.g., to allow simulations), including their features, benefits, and limitations. It is also recommended that organizations adopt well-defined Success Management processes (e.g., Varajão et al. (2022a)). We strongly believe that Tables 1 and 2, listing the extant models/methods, together with Tables 4 and 5, identifying the main benefits and limitations of each model/method, can be valuable in this process.

Figure 3.

Selecting/defining a model/method for evaluating the success of a specific project.

6. Conclusion

Evaluating an IS project’s success is a complex task because of the many perceptions about success, depending, for example, on the stakeholders, project characteristics, project management characteristics, and many other aspects. Therefore, adopting a model/method for the project evaluation is not a simple or elementary task.

We carried out a literature review aiming to identify and raise awareness of the existing models/methods to evaluate IS projects' success. It also included a review of models/techniques from non-IS projects since some may be suitable for IS projects.

One limitation of this study is related to other models/methods that may exist and that were not considered due to not being the focus of research or published in scientific outlets. Another limitation regards the benefits and limitations of the models/methods since the ones presented in this article are reported by the original authors. A description of each technique would also be useful. However, that was not possible to include it in this paper due to length limitations (moreover, they are available in the original sources). All of these limitations create paths for research.

The listing of models/methods, and discussion of its characteristics, current applications, benefits, and limitations, are the main contributions of this article. For researchers, it provides several insights into the state-of-the-art and helps to identify new avenues for research. For practitioners, it improves the understanding of the role of the evaluation models/methods. It also provides a basis for selecting the appropriate models/methods for particular projects according to their characteristics.

Declarations

Author contribution statement

All authors listed have significantly contributed to the development and the writing of this article.

Funding statement

This work was been supported by FCT - Fundação para a Ciência e Tecnologia within the R&D Units Project Scope (UIDB/00319/2020).

Data availability statement

No data was used for the research described in the article.

Declaration of interest’s statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.

Appendix.

The search strings are listed next.

IDENTIFIER	EXPRESSION
Exp1	TITLE(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION”) AND TITLE(PROJECT) AND TITLE(“PERFORMANCE” OR “SUCCESS” OR “ATTAINMENT” OR “ACCOMPLISHMENT” OR “ACHIEVEMENT” OR “REALISATION” OR “REALIZATION”)
Exp2	TITLE(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION” OR “EVALUATING” OR “ASSESSING” OR “APPRAISING” OR “ESTIMATING” OR “CALCULATING”) AND TITLE(“PROJECT” OR “PROJECTS”) AND TITLE(“PERFORMANCE” OR “SUCCESS” OR “ATTAINMENT” OR “ACCOMPLISHMENT” OR “ACHIEVEMENT” OR “REALISATION” OR “REALIZATION”)
Exp3	TITLE(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION” OR “EVALUATING” OR “ASSESSING” OR “APPRAISING” OR “ESTIMATING” OR “CALCULATING”) AND TITLE(“PROJECT” OR “PROJECTS”) AND TITLE(“PERFORMANCE” OR “SUCCESS” OR “ATTAINMENT” OR “ACCOMPLISHMENT” OR “ACHIEVEMENT” OR “REALISATION” OR “REALIZATION”) AND KEY(“MULTIPLE CRITERIA”)
Exp4	TITLE(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION”) AND TITLE(PROJECT) AND TITLE(“PERFORMANCE” OR “SUCCESS” OR “ATTAINMENT” OR “ACCOMPLISHMENT” OR “ACHIEVEMENT” OR “REALISATION” OR “REALIZATION”) AND TITLE(“METHOD” OR “TECHNIQUE” OR “SYSTEM” OR “PROCEDURE”)
Exp5	TITLE(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION” OR “EVALUATING” OR “ASSESSING” OR “APPRAISING” OR “ESTIMATING” OR “CALCULATING”) AND TITLE(“PROJECT” OR “PROJECTS”) AND TITLE(“PERFORMANCE” OR “SUCCESS” OR “ATTAINMENT” OR “ACCOMPLISHMENT” OR “ACHIEVEMENT” OR “REALISATION” OR “REALIZATION”) AND TITLE(“METHOD” OR “TECHNIQUE” OR “SYSTEM” OR “PROCEDURE” OR “PROCESS”)
Exp6	TITLE(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION”) AND TITLE (“PERFORMANCE OF PROJECT” OR “SUCCESS OF PROJECT” OR “ATTAINMENT OF PROJECT” OR “ACCOMPLISHMENT OF PROJECT” OR “ACHIEVEMENT OF PROJECT” OR “REALISATION OF PROJECT” OR “REALIZATION OF PROJECT” OR “PROJECT PERFORMANCE” OR “PROJECT SUCCESS” OR “PROJECT ATTAINMENT” OR “PROJECT ACCOMPLISHMENT” OR “PROJECT ACHIEVEMENT” OR “PROJECT REALISATION” OR “PROJECT REALIZATION”)
Exp7	KEY(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION”) AND KEY(“PERFORMANCE OF PROJECT” OR “SUCCESS OF PROJECT” OR “ATTAINMENT OF PROJECT” OR “ACCOMPLISHMENT OF PROJECT” OR “ACHIEVEMENT OF PROJECT” OR “REALISATION OF PROJECT” OR “REALIZATION OF PROJECT” OR “PROJECT PERFORMANCE” OR “PROJECT SUCCESS” OR “PROJECT ATTAINMENT” OR “PROJECT ACCOMPLISHMENT” OR “PROJECT ACHIEVEMENT” OR “PROJECT REALISATION” OR “PROJECT REALIZATION”)
Exp8	ABS (“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION”) AND ABS (“PERFORMANCE OF PROJECT” OR “SUCCESS OF PROJECT” OR “ATTAINMENT OF PROJECT” OR “ACCOMPLISHMENT OF PROJECT” OR “ACHIEVEMENT OF PROJECT” OR “REALISATION OF PROJECT” OR “REALIZATION OF PROJECT” OR “PROJECT PERFORMANCE” OR “PROJECT SUCCESS” OR “PROJECT ATTAINMENT” OR “PROJECT ACCOMPLISHMENT” OR “PROJECT ACHIEVEMENT” OR “PROJECT REALISATION” OR “PROJECT REALIZATION”)
Exp9	TITLE(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION" OR “CALCULATION”) AND TITLE(“PERFORMANCE OF PROJECT” OR “SUCCESS OF PROJECT” OR “ATTAINMENT OF PROJECT” OR “ACCOMPLISHMENT OF PROJECT” OR “ACHIEVEMENT OF PROJECT” OR “REALISATION OF PROJECT” OR “REALIZATION OF PROJECT” OR “PROJECT PERFORMANCE” OR “PROJECT SUCCESS” OR “PROJECT ATTAINMENT” OR “PROJECT ACCOMPLISHMENT” OR “PROJECT ACHIEVEMENT” OR “PROJECT REALISATION” OR “PROJECT REALIZATION”) AND TITLE(“INFORMATION SYSTEM” OR “INFORMATION SYSTEMS” OR “INFORMATION TECHNOLOGY” OR “INFORMATION TECHNOLOGIES” OR “ICT” OR “IT/IS” OR “IS/IT”)
Exp10	ABS(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION”) AND ABS(“PERFORMANCE OF PROJECT” OR “SUCCESS OF PROJECT” OR “ATTAINMENT OF PROJECT” OR “ACCOMPLISHMENT OF PROJECT” OR “ACHIEVEMENT OF PROJECT” OR “REALISATION OF PROJECT” OR “REALIZATION OF PROJECT” OR “PROJECT PERFORMANCE” OR “PROJECT SUCCESS” OR “PROJECT ATTAINMENT” OR “PROJECT ACCOMPLISHMENT” OR “PROJECT ACHIEVEMENT” OR “PROJECT REALISATION” OR “PROJECT REALIZATION”) AND ABS(“INFORMATION SYSTEM” OR “INFORMATION SYSTEMS” OR “INFORMATION TECHNOLOGY” OR “INFORMATION TECHNOLOGIES” OR “ICT” OR “IT/IS” OR “IS/IT”)
Exp11	TITLE(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION”) AND TITLE(“PROJECT” OR “PROJECTS”) AND TITLE(“PERFORMANCE” OR “SUCCESS” OR “ATTAINMENT” OR “ACCOMPLISHMENT” OR “ACHIEVEMENT” OR “REALISATION” OR “REALIZATION”) AND TITLE(“INFORMATION SYSTEM” OR “INFORMATION SYSTEMS” OR “INFORMATION TECHNOLOGY” OR “INFORMATION TECHNOLOGIES” OR “ICT” OR “IT/IS” OR “IS/IT” OR “INFORMATION AND COMMUNICATIONS TECHNOLOGY”)
Exp12	KEY(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION”) AND KEY(“PROJECT” OR “PROJECTS”) AND KEY(“PERFORMANCE” OR “SUCCESS” OR “ATTAINMENT” OR “ACCOMPLISHMENT” OR “ACHIEVEMENT” OR “REALISATION” OR “REALIZATION”) AND KEY(“INFORMATION SYSTEM” OR “INFORMATION SYSTEMS” OR “INFORMATION TECHNOLOGY” OR “INFORMATION TECHNOLOGIES” OR “ICT” OR “IT/IS” OR “IS/IT” OR “INFORMATION AND COMMUNICATIONS TECHNOLOGY”)
Exp13	TITLE(“EVALUATION” OR “ASSESSMENT” OR “APPRAISAL” OR “VALUATION” OR “ESTIMATION” OR “CALCULATION” OR “EVALUATING” OR “ASSESSING” OR “APPRAISING” OR “ESTIMATING” OR “CALCULATING”) AND TITLE(“PROJECT” OR “PROJECTS”) AND TITLE(“PERFORMANCE” OR “SUCCESS” OR “ATTAINMENT” OR “ACCOMPLISHMENT” OR “ACHIEVEMENT” OR “REALISATION” OR “REALIZATION”) AND TITLE(“INFORMATION SYSTEM” OR “INFORMATION SYSTEMS” OR “INFORMATION TECHNOLOGY” OR “INFORMATION TECHNOLOGIES” OR “ICT” OR “IT/IS” OR “IS/IT” OR “INFORMATION AND COMMUNICATIONS TECHNOLOGY” OR “INFORMATION AND COMMUNICATIONS TECHNOLOGIES”)
Exp14	TITLE(“EVALUAT∗” OR “ASSESS∗” OR “APPRAIS∗” OR “MEASUR∗” OR “VALUAT∗” OR “CALCULAT∗”) AND TITLE(“PROJECT∗”) AND TITLE(“PERFORM∗” OR “SUCCESS” OR “ATTAIN∗” OR “ACCOMPLISH∗” OR “ACHIEVE∗” OR “REALIS∗” OR “REALIZ∗”)
Exp 15	TITLE(“EVALUAT∗” OR “ASSESS∗” OR “APPRAIS∗” OR “MEASUR∗” OR “VALUAT∗” OR “CALCULAT∗”) AND TITLE(“PROJECT∗”) AND TITLE(“PERFORM∗” OR “SUCCESS” OR “ATTAIN∗” OR “ACCOMPLISH∗” OR “ACHIEVE∗” OR “REALIS∗” OR “REALIZ∗”) AND TITLE(“METHOD∗” OR “PROCESS” OR “TECHNIQUE” OR “APPROACH”)