Evaluation of Information Systems Project Success – Insights from Practitioners

ABSTRACT Evaluating the success of projects should be a key process in project management. However, there are only a few studies that address the evaluation process in practice. In order to help fill this gap, this paper presents the results of an exploratory survey with experienced information systems project managers. Results show that opportunities for lessons learned and project management improvement are being missed due to the lack of formal evaluation of success.


Introduction
In an increasingly technological world, information systems (IS) provide support at several business levels, from operational transactions to strategic initiatives. On the one hand, achieving success in IS projects is fundamental to organizations (Varajão, 2018a). On the other hand, evaluating success can be a huge endeavor because, due to the complexity of projects, there are many variables to take into account (Morcov et al., 2020) which are of a technical, behavioral, and contextual nature.
The evaluation of success in projects is a subject of research for a long time (Aragonés-Beltrán et al., 2017;Liang et al., 2017;Osei-Kyei & Chan, 2018). Several works suggest that project success should be evaluated throughout the project lifecycle (Teixeira et al., 2019;Varajão, 2016), as well as in the post-project (Karlsen et al., 2005;Varajão, 2018a;Varajão & Trigo, 2016). Besides, since the criteria for success evaluation are particular to each project (Albert et al., 2017), to define a process for success management (focused on the understanding, evaluation and reporting of success) is fundamental (Varajão, 2018b).
However, in the literature, there is limited evidence on actions or criteria actually used by practitioners to assess the success of IS projects. Furthermore, several studies (e.g., Albert et al. (2017) and Varajão and Carvalho (2018)), also show that in practice the evaluation of success is many times not performed through systematic processes. The consequences of not evaluating the success of a project may result in the waste of efforts and resources (Pujari & Seetharam, 2015).
Aiming to help fill this gap, an exploratory survey was carried out with experienced IS project managers. Our contribution concentrates on three major questions. The first relates to the moment in the project when the process for evaluating success is defined and also who participates in that process. The second relates to the criteria for evaluating success (namely, which criteria are defined in projects, how and by whom). The third regards to when and how the project success is measured.
The article is structured as follows: the next section presents a brief literature review on project management and IS project success; the third section describes the theoretical framework and the research method; the fourth section presents the survey results; the fifth section presents the discussion of the results; and finally, the last section has conclusions, main contributions, and prospects for further work.

Project and project management
According to Munns and Bjeirmi (1996) and the PMI (2017), a project can be understood as the search for the achievement of an objective, involving tasks and the use of resources. Projects must be carried out following specifications, having a well-defined beginning and end. It is commonly accepted that a project is a unique effort, i.e., a special endeavor that has not been previously done (Dvir et al., 2003).
Regardless of the various dimensions, forms, degrees of risk, complexity of the resulting products or services, projects are the focus of project management. Catarino et al. (2009) refer that project management is a vast area of knowledge that helps project managers and their teams to plan, execute, monitor, and control the tasks of the projects in which they are involved. Munns and Bjeirmi (1996) state that project management involves defining requirements, planning execution, allocating resources, monitoring progress, and adjusting deviations from the project plan. Project management involves managing the schedule, cost, scope, quality, risk, communications, resources, acquisitions, stakeholders, and integrating all of the above (PMI, 2017). In particular, IS project management consists of applying project management to IS projects in an organization (Booth & Philip, 2005).
To understand the success of an IS project, awareness of its context and specificities is needed. Different types of projects have distinct strategic importance, which typically requires different management approaches (Artto & Dietrich, 2007). Cadle and Yeates (2008) identify nine types of IS projects that cover most projects: software development; package implementation; system enhancement; consultancy and business analysis assignments; systems migration; infrastructure implementation; outsourcing (and in-sourcing); disaster recovery; and smaller IS projects.
IS projects, such as digital transformation projects, typically involve changes in business and ways of working of an organization, which are caused by the adoption of digital technologies (Kääriäinen et al., 2020). Examples of technologies that organizational IS projects typically involve are (Trigo et al., 2007;Varajão et al., 2009): Enterprise Resource Planning (ERP); Business Intelligence (BI); Customer Relationship Management (CRM); Supply Chain Management (SCM); Collaboration and Groupware (CG); and Workflow Management Systems (WMS). Considering that there are different kinds of IS projects, guiding the IS area to success is a complex task since there are many variables to be taken into account, such as the need to manage various stakeholders (e.g., IS users) during the project (Santos & Varajão, 2015).
Guides and standards for project management are valuable in this context, such as PMBOK (PMI, 2017), PRINCE2 (AXELOS, 2017), RIBA plan of work (RIBA, 2020), ICB-IPMA Competence baseline (IPMA, 2016), PM2 (EU, 2018), and APM Body of Knowledge (APM, 2019). In addition to project management standards, there are also other guides, such as maturity models (e.g., Capability Maturity Model Integration (CMMI, 2018) or Organizational Project Management Maturity Model (OPM3) (PMI, 2013)). The main objective of maturity models is to help to understand the level of management capability of an organization regarding project management and to identify opportunities for development. They have in common the idea that the higher the level of maturity, the greater the performance of project management (White & Fortune, 2002). The experience, the process, and the level of maturity of project management are aspects that can impact success in projects.

Success in IS projects
Success may have several meanings, depending on the perspective and the stakeholder. Freeman and Beale (1992) mention that success means different things to different people: "For an architect, success is considered in terms of esthetic appearance, for an engineer in terms of technical competence, for an accountant in terms of money spent on a budget, for a human resources manager in terms of employee satisfaction, and a Chief Executive Officer (CEO), for example, can evaluate success considering the market share". The concept of success is broad and must include all stakeholders' perceptions (Wateridge, 1998). Stakeholders are, for example, customers, developers, the project team or the end-users (Shenhar et al., 2001).
There are two perspectives that should be highlighted regarding the success of a project (Shenhar et al., 2001(Shenhar et al., , 1997. One perspective is that all business objectives can be met, even if project management still fails to be in compliance with schedule, budget and quality. Another perspective is that project management can be successful, for example, in terms of scope, quality and budget, but the project does not meet the client's expectations. On one side, there is the project management; on the other side, there are the expected benefits. Determining the success of a project can be ambiguous and difficult to measure, as some evaluation criteria can be met, and others may not. In other words, success may not always be achieved in all its aspects, and it can be partial (Baccarini, 1999). Of course, if a project, in spite of complying with scope, schedule and budget, does not satisfies the customer, hardly can be considered a total success. Dvir et al. (2003) highlight three aspects of the project's performance as reference points for measuring the success (or failure) of a project: the implementation process; the perceived value of the project; and customer satisfaction with the delivered project. According to Shenhar et al. (1997) there are four dimensions of success: • Project efficiency: It is a short-term metric that measures the efficiency with which each project process was managed. It is an indicator that the project was completed on time and on budget. However, it is a measure that only indicates good project management, but it does not help to measure success in the long term. It is an important dimension because the increasing competitiveness, short life cycles and the increasingly reduced timeto-market, makes it a critical component for competitiveness; • Impact on the customer: This dimension is related to the customer and/or the end-user. This measure addresses the importance of compliance of needs and requirements defined by them. The customer satisfaction can be related to new follow-up projects, or new versions of the same product, or results that are important to the success of the organization; • Business success: This dimension concerns the impact that the project can have on the organization, for instance, whether the project generated sales, profits, business results, or more market share; • Preparation for the future: Dimension that regards to the contribution to the future of the organizational and technological structure. It is a long-term dimension that involves preparing future opportunities, exploring new markets, new ideas, innovations and products, generating new essential skills and competencies. Baccarini (1999) identifies two dimensions of success: the "hard" and "soft" dimensions. The "hard" dimension refers to criteria such as cost, schedule, and quality. These are easier to measure, and it is also relatively easy to reach a consensus on these criteria. The dimension of the "soft" criteria refers to aspects such as happiness, job satisfaction, improved reputation, and attention to detail. The later dimension is more difficult to assess. Freeman and Beale (1992) found seven main criteria for measuring success: technical performance; efficiency of project execution; managerial and organizational implications (client and user satisfaction); personal growth; project termination; technical innovativeness; manufacturability; and business performance. Atkinson (1999) states that over many years the well-known Iron Triangle (cost, time and quality) was linked to the very definition of successful project management. Subsequently, he presents the Square Route, stating that the Iron Triangle excluded long-term aspects that should be present in the success criteria. The Square Route includes the Iron Triangle itself, the IS, organizational benefits, and benefits related to stakeholders. Karlsen et al. (2005) present as one of the central questions, "when should the degree of success of the project be evaluated?". They recommend that the assessment should include all criteria covering the project processes, the project results, the use of the project results, and the effects of the project results. The criteria must be defined as early as possible in the project; however, it is necessary to take into account the changes that occur during the project, which can change the success criteria. They also suggest that space for interaction and sharing of opinions and expectations by all interested parties should be created by the project managers. And, finally, they state that the evaluation should be made in the project delivery phase and during the post-delivery phase of the project, to have a complete picture of the project users' vision and the effects that the project will have on the client organization. Varajão (2016Varajão ( , 2018b proposed success management as a new knowledge area of project management as well as a systematic process, including activities to plan, evaluate, monitor, and report the success. For evolving the understanding of success, it is important to differentiate between success factors and criteria. Westerveld (2003) explains this difference by mentioning that, to manage a project successfully, it is necessary to focus on result areas (related to project success criteria) and organizational areas (related to success factors). Success factors are related to circumstances, facts, and influences (Lim & Mohamed, 1999). Success criteria are the dependent constructs by which the project will be judged as successful (or not) (Turner, 2014). Examples of success criteria are customer and project team satisfaction, compliance with cost, time, and scope (Osei-Kyei et al., 2017). Some success factors are project manager experience, team capability, and top management support (Gunduz & Almuajebh, 2020). As our research focuses on the success evaluation, the success criteria will be addressed in the next sections.
Focusing on the particular case of IS, Delone and McLean (2003) present the D&M IS Success Model, which is a seminal work that identifies six relevant constructs for IS success: System quality; Information quality; Service quality; Intention to use/Use; User satisfaction; Net impacts. This work addresses the criteria (post-project), but not the evaluation process, which is currently a gap in the IS literature.

Method
This section presents the theoretical framework and the underlying research questions, as well as the survey, data collection and analysis procedures.

Theoretical framework
Establishing a process for evaluating success, measuring it consistently, and using the measurement results to manage the project are good practices for achieving better results (Thomas & Fernández, 2008). Being important to study this process in the case of IS projects, the theoretical framework and research questions defined for assessing success evaluation practices are shown in Figure 1. It is based on the work of Varajão (2016Varajão ( , 2018b, assuming that success management includes the definition of the evaluation process, the identification of criteria to evaluate success, and the evaluation and reporting of success. The measurement instrument described in section 3.2 was defined following the theoretical framework and the corresponding research questions. Defining the evaluation process is an important step toward an accurate definition of success (Varajão, 2016(Varajão, , 2018b) (RQ1.1). For the process to be effective, it is essential to understand when and by whom it is defined (Wateridge, 1998) (RQ1.2), since the success perspective is dependent on the stakeholders (Nelson, 2005). In other words, for evaluating success it is needed to define criteria (Atkinson, 1999;Ika, 2009;Wateridge, 1998) considering who should take part in that definition (RQ2.x).
The consideration of the moments when the evaluation of success is carried out and who participate in the evaluation are also important to better understand the process (RQ3.x) (Karlsen et al., 2005) as well as its effects. All these aspects need to be taken into account in a success management process .
To get answers to these questions from the field, a survey was carried out with experienced IS project professionals. Aiming to gain a deeper insight into the "how" and "why" of some of the obtained results, we also carried out qualitative in-depth interviews. There are important aspects related to success that have been addressed by research; however, little is known about the daily practice, which is the focus of our study.

Data collection and analysis
A survey was conducted to collect data for this research. Surveys are information collection methods used to describe, compare, or explain individual and societal knowledge, feelings, values, preferences, and behavior (Fink & Kosecoff, 1985). Surveys have a wide variety of aims, such as collecting public opinion, with the majority of surveys targeting administrative, commercial, or scientific purposes. Data can be collected in many ways, including by phone, e-mail, online, or directly by interview. The results of surveys must be presented in completely anonymous summaries, for assuring the confidentiality of the data provided by the respondents. Scheuren (2004) refers that the word survey is often used to describe a method of collecting information from a sample of a population of individuals. In our study, the population is composed of IS project managers.
Our survey was questionnaire-based. The elaboration of a questionnaire is one of the most critical steps in the process of developing a survey (Scheuren, 2004). It should start with the objectives of data collection. The objectives must be well defined and, subsequently, the specific information needed to satisfy the objectives RQ  must be identified. The communication channel for data collection must be decided at the beginning of the process. The questions can be closed-ended or open-ended. However, some closed-answer questions can influence how people respond to the question. Fink and Kosecoff (1985) refer to a questionnaire as being the main instrument for collecting data by sampling.
For the elaboration of the questionnaire, an online platform was used (Google Forms). In the process of selecting the platform, a user-friendly tool was sought, with the objective of reducing the learning curve for the elaboration and response to the questionnaire. In the questionnaire, there were closed and open questions, aiming to collect perceptions and insights from managers regarding the questions identified in the research framework ( Figure 1). Issues such as the length of the questionnaire applied, the complexity and sensitivity of the issues were taken into account.
Before carrying out the survey, to ensure readability and validity, a pretest was conducted with two researchers and an IS project manager, whose feedback was requested regarding the sequence of questions, as well as the clarity and objectivity of the used language. Their feedback has been incorporated in the final version of the questionnaire (presented in Appendix A), which required only minor rewording.
A sample is a fraction of the population, and the sample size depends on the purpose of the study. With the questionnaire and the universe (IS project managers) defined for this research, the snowball approach was firstly used to collect data. According to Harrell and Bradley (2009), snowball sampling occurs when a participant suggests another participant to the investigator. They also emphasize that snowballing can be dangerous for research, as sampling may represent only a single group in a network. Rea and Parker (2014) state that snowball sampling is beneficial when it is difficult to identify potential respondents. When some respondents are identified, they are asked to identify other possible participants.
For our study, as the research progressed, the snowball collection method has shown to be insufficient, due to a low number of participants. The research team opted to complement this method by delivering the questionnaire by phone. The expectation was that through direct contact, more respondents would be available. After contacting around 100 companies and realizing that the responses are still not evolving as expected, it was decided to contact project management associations to advertise the study within their communities, such as APOGEP (the Portuguese association of project management connected to the International Project Management Association) and the Portuguese chapter of the Project Management Institute (PMI). After the contact with the associations, there was a small acceleration in the number of responses. However, that number quickly stagnated again.
Since all methods were being very slow in terms of dissemination and the responses obtained were not appearing as expected, it was decided to disseminate the questionnaire on social networks associated with project management groups (e.g., LinkedIn). After this long effort, the final result of the sample was 64 valid responses. All participants have experience in project management, with a diverse and complementary view of IS projects. One possible reason for the difficulty getting participants may be the low maturity of companies regarding the evaluation of success, which inhibited project managers from answering the questions related to this subject.
After analyzing the survey's results, to complement them and gain deeper insights, we carried out qualitative in-depth interviews with IS project management experts.

Results
This section presents the practices related to the evaluation of success in IS projects, as reported by survey participants. In alignment with the research framework, the studied practices involve the definition of the process for evaluating the success, the stakeholders' participation in the process, criteria for assessing success, the ways for obtaining information and evaluating success during the project life cycle, the overall results regarding the levels of success achieved, as well as the respondents' opinion regarding the success assessment practices considered most advantageous for their organizations.

Demographic information
Table 1(a) summarizes the demographics of participating project managers. Half of the participants are more than 40 years old (50%). Regarding experience in project management, 45.31% have more than ten years of experience, and 43.75% have managed more than 20 projects. About 53.13% of the respondents indicated that they held training/certification in project management. It should be noted that 46.87% of the participants reported never having received training or certification in project management. It is an indicator that there is still a long way to go in this area, concerning training to improve and deepen knowledge and competencies.
Table 1(b) summarizes the characteristics of the respondents' companies. Respondents came from organizations of different sizes (small, medium and large).
The majority of companies have an international presence (82.81%). Many companies have their project management methodology aligned with PMBOK (37.50%), and only 15.62% use a project management maturity model, suggesting that maturity models are not commonly used to improve project management. Table 2 shows the distribution of the participants' companies, according to the sectors of activity in which they operate. What stands out most is Engineering, with 25% of the responses. It should be noted that around 39% of the respondents answered "Other sectors", referring in these cases mainly to "Telecommunications", "Information Technologies", "Information Systems", "Aeronautics" and "Shared services".
To sum up, the respondents are experienced project managers, representing a wide variety of company sizes and project management approaches.

Definition of the evaluation process
In the context of this study, the evaluation process is defined as a set of activities (and related aspects) implemented to evaluate and report the success of a project (Varajão, 2016). Table 3 shows that, when the evaluation process is defined, in around 43% of cases, it is defined before the project starts, mainly during the project initiation or project planning. In 35.93% of cases, it is not formally defined, which is a matter of concern ("if you cannot measure it, you cannot manage it").

Definition of criteria to evaluate success
Regarding the definition of the success criteria, Table 4 shows that in 31.25% of the cases there is a predefined   list of criteria and in 20.31% of the cases the criteria are discussed with the stakeholders. In 43.75% of the cases, the criteria are not defined formally. This result is interesting since 35.93% of the respondents say that the evaluation process is not formally defined (see Table 3). This suggests that in about 8% of those cases, the process is defined, but is mainly informal. Table 5 shows the criteria that are defined to evaluate success. In the large majority of cases, the criteria are the ones from the well-known "Iron triangle" (time compliance (60.94%), scope compliance (60.94%), cost compliance (54.69%)), followed closely by client/customer satisfaction (50%).
It is possible to note that, in addition to the more classic criteria (scope, quality, time, and cost), several criteria related to stakeholders' satisfaction (client/customer, users, executing team, sponsor, others) were mentioned, as well as criteria that are beyond the success of project management, such as "Contribution for the development of the organization", "Preparation for the future," or "Social impact". The introduction of new criteria to assess success reveals a evolution in this area, showing that managers slowly start to be concerned with the business benefits and not just with the efficiency of projects.

Participants in the evaluation process and definition of criteria to evaluate success
As shown in Table 6(a), in the majority of cases, the project manager participates in the definition of the evaluation criteria. However, the client and other stakeholders only participate in 32.81% or less of the cases. As shown in Table 6(b), in 68.75% of the cases, the project manager participates formally in the evaluation process, the client participates in 45.31% of the cases, and the sponsor in 43.75%. PMO, management team, and executing team are also participating in the definition of criteria and evaluation of success.

Success evaluation
Regarding the moments when the success is evaluated in the project, Table 7 shows that in 65.63% of the cases this occurs only at the project closing. In 23.44% of the cases, it is not evaluated at all. Considering that in 35.93% of the cases, the evaluation process is not defined (see Table  3), it means that in several cases, the success is evaluated   only informally. One (!) of the respondents mentioned that success is also evaluated post-project. Information for evaluating success can be collected in several ways. As shown in Table 8, most times, the information is collected in meetings (45.31%) and reports (45.31%). The resulting deliverables are also commonly used as a basis for the success evaluation (in 35.94% of the cases).

Overall results of project management
For a better insight of project management results at the participating companies, it was requested to the respondents to report the last projects they participated as project managers. Each participant gave information about one (minimum) to three (maximum) completed projects, resulting in a total of 158 projects. We only considered projects reported with complete details (ten projects were excluded because participants did not provide budget information). Table 9 shows the reported project types (44.3% of the projects were focused on custom software development).
Regarding the budget, the majority of the surveyed projects had a budget below 100 K€ (56.08%), with a duration of fewer than 7 months (54.43%).
Considering the results obtained, as shown in Table 10, in overall, the projects were well succeeded, but only 10.76% were classified as a total success (level 10). No projects were abandoned (level 0) in this sample.

The opinion of the participants regarding the evaluation process
The questionnaire included an open question to get the opinion of the project managers regarding the evaluation aspects they think would be advantageous to put in practice in their companies, as shown in Table 11. By looking at the answers, it stands out that many mentioned aspects are related to the evaluation process and criteria for evaluation.

Discussion
The majority of respondents have extensive experience in IS projects. Almost two-thirds managed at least ten projects and more than 45% have more than 10 years of experience. The international experience should also be highlighted since more than 80% of the companies where they work have an international presence.

Definition of the evaluation process (RQ1)
The definition of the evaluation process is essential for assessing the success of IS projects. The process for evaluating the success should be formally defined and the success evaluated along with all the project lifecycle (Teixeira et al., 2019). Regarding RQ1.1 ("when is defined the process for evaluating the success?"), in about 43% of cases, the process is defined before the executing phase (4.69% before the project, 21.88% do it in the project initiation phase, and 17.19% in the project planning phase). The delivery of the products/services was also cited as a moment for the definition of the evaluation of the success process, with 1.56%. The phases of execution and closing of the project life cycle have a total of 12.5%, being equally proportional in this research.
Almost 36% of respondents say that they do not have formal success assessment processes and more than 6% define the evaluation process at any time during the project (Table 3). To sum up, in more than 42% (35.93 not formally plus 6.25 any time) of the cases, managers and project teams may not have a structured process for evaluating success. In our sample, more than half of the projects surveyed had a budget of less than 100 K€ (56.08%) and a duration of fewer than 7 months (54.43%), which may bias results. However, research conducted by Varajão and Carvalho (2018) in 10 companies of various sizes (from 14 to 400,000 employees) has shown similar results, leading to the conclusion that "the evaluation of project success is currently an informal and rudimentary process mainly focused on the success of project management and not on the success of the projects' deliverables" and related benefits. Regardless of the size of the project, the many aspects related to success evaluation should be defined early in the project as shown by research (e.g., (Takagi & Varajão, 2019;Teixeira et al., 2019)); however, this seems to be not happening in daily practice.
The client and the project team are important stakeholders in the process of evaluating success. Regarding RQ1.2 (who takes part in the process?), participants who normally intervene are the project manager (68.75%), the client (45.31%), the project sponsor (43.75%), the project execution team (35.94%), the project management team (34.38%), and the PMO (28.13%). This shows that the client participates in the project less than 50% of the times. Since customer satisfaction should be one of the main criteria for evaluating success (Atkinson, 1999;Baccarini, 1999;Westerveld, 2003), this low participation of the client is surprising and may be a deterrent to the success of projects.

Definition of criteria to evaluate success (RQ2)
According to Atkinson (1999), it is necessary to use different success criteria besides the Iron Triangle. Nevertheless, our survey concludes that the so-called Iron Triangle still continues to be popular nowadays (as it was expected). In other words, time, cost, scope, and quality are very present when assessing the success of a project, together with customer satisfaction, which is important because without the customer's approval regarding the resulting products/services the project cannot achieve the desired success. Responding to RQ2.1 ("what criteria are used to evaluate success?"), the criteria formally used to assess the success of the projects that stand out are: time (60.94%); scope and requirements (60.94%); budget (54.69%); customer satisfaction (50%); and product/service quality (34.38%). In several cases, these are the only criteria defined. Regarding other criteria to note are the project's business objectives, along with quality (34.38%). Marques et al. (2013) indicate that the three main criteria typically used to assess the success of software development projects are compliance with the budget, compliance with deadlines, and compliance with the scope. Although these continue to be extremely important, the success should be measured based on a richer and broader set of aspects (Paiva et al., 2011). However, we also conclude that compliance with the budget, deadline, and scope remains the criteria most common for evaluating project management success, as our results also show. To note that, regardless of its importance, customer satisfaction is defined in only half of the cases. Clients' business objectives and user satisfaction are the next in the ranking (18.75%). User satisfaction is one of the most-cited criteria in the IS/IT literature (Iriarte & Bayona, 2020). The importance of user satisfaction can be derived from the acceptance of IS success models, such as those proposed by DeLone and McLean (1992), Delone & McLean (2003), and Bradford and Florin (2003). The sponsor's satisfaction and the project team's satisfaction had a percentage of 14.06%, followed by "organizational development." Criteria related to sponsor and team satisfaction are important elements to measure the success of the project (Westerveld, 2003).
Regarding the definition of the criteria for assessing success, most participants responded that there is no formal definition of the criteria (43.75%). As noted earlier, most companies did not formally define the success assessment process, so the formal definition of the criteria does not happen either. However, of those who define the criteria, 31.25% say that there is a predefined list of criteria, which is discussed and enriched together with the stakeholders; 20.31% state that the evaluation criteria are defined together with the interested parties, in each project; and 4.69% say that there is a predefined list of criteria, which is not discussed with stakeholders. Those who participate most in the definition of the criteria are the project managers (71.88%); customers (32.81%); the PMO (32.81%); the project sponsor (31.25%); and the project management team (28.13%). According to Varajão and Trigo (2016), the definition of a predefined list of evaluation criteria is useful but should be adapted to each project. These results respond to RQ2.2 ("how are defined the criteria?") and RQ2.3 ("who takes part in the definition of criteria?").

Perform success evaluation (RQ3)
Regarding RQ3.1 ("when take place the evaluation actions to measure success?"), this research shows that most projects are evaluated at the end (65.63%); 31.25% is evaluated more than once during its execution; 4.69% are assessed at the beginning; 1.56% only once during its execution; and in 23.44% of the projects, success is not evaluated. In IS projects it is common that the deliverables are deployed not only at the end of the project, but also several times during the project (e.g., in agile projects). Thus, it would be expected that the evaluation occurred during the project more frequently.
The information for the evaluation of a project can be obtained in several ways (for instance, workshops, joint meetings, interviews, management reports, surveys, stakeholders' testimonies, among others) and must be combined to enrich the perception of success (Varajão & Trigo, 2016). Considering RQ3.2 ("how is collected the information for the evaluation?") and according to the project managers participating in the study, the collection of information to assess success is usually mainly done by two means: through execution reports and joint meetings, both with 45.31% participants mentioning it.
Results show that there are many projects (more than 35%) that have not formally defined processes and criteria for evaluating success. This in line with Varajão and Carvalho (2018) who report that evaluation of success is typically ad-hoc or informal in IS projects. However, on a scale of 0 to 10, just about 15% of the reported projects are between 0 (abandoned project) and 6. Most projects (more than 60%) are between 8 and 10 (complete success), according to the respondents. If we set level 7 as the threshold of success, the result is similar to that found by Robic and Sbragia (1995) in IS projects. However, it should be mentioned that the reported levels of success, in many cases, are the result of perceptions and not of formal evaluations.

Insights from the respondents
A final question was asked to the survey's participants: "Part of the success or failure that sometimes happens in IS projects may be due to the lack of systematic success assessment practices. Do you agree with this statement?". Since the confidentiality of responses was assured, it was possible to obtain open and spontaneous perspectives, which better reflect the reality.
The majority of participants (87.8%) agreed with the statement, adding that often the failure of projects is also due to "resistance to change", "lack of rigor in product control", "communication problems", "poor planning", "poor management of multidisciplinary teams", and "poor management of stakeholders". They mentioned that: "Often the scope of the project is incorrect in relation to the expectations of customers and stakeholders. It is crucial to constantly monitor the project so that, if any deviation is noticed, it is less hard to put it on the path to success." "A constant evaluation, from the beginning of the project, is essential because it allows addressing serious anomalies and, to avoid customer dissatisfaction." "Without evaluation, there is no possible learning and continuous improvement." "Yes, I agree. The measurement, involving data collection, provides valuable information on the condition of the project, with the aim of ensuring that the objectives are being achieved. Organizations must constantly measure, monitor, and control, seeking to continuous performance improvement. In case of interaction with the customer, the analysis of the processes must also be evaluated periodically, to determine the impact of the IS on the actions and behaviors of the customers. It is necessary to share the analysis of results with managers related to the implementation of the SI. I believe that, without this type of practice is not possible to classify a project as successful or not, regardless of the different variables and particularities that each IS project may have." Other participants mentioned that time-to-market often makes it difficult to invest in these types of processes, which reflects the current state of the market due to strong competition: "Many organizations speed up their tasks, processes, and activities so that the product launch is done as quickly as possible, getting competitive advantage." This leads organizations to reduce some of the management processes, tasks, and activities. According to several participants, it often means that processes related to project evaluation, monitoring and lessons learned are sacrificed. Also noteworthy are the responses that warn about the insufficiency of systematization and formalization of success assessment practices. It has been indicated that the evaluation criteria arise at a very advanced stage in the life cycle of the projects, and: "The evaluation of the execution of the projects is systematically confused with the evaluation of the deliverables." Respondents who did not agree with the statement added that: "The problems are other", such as, "poor change management, the definition of the incorrect scope on stakeholder expectations, failure in the execution phase and lack of rigor in control." Others indicate that failure also depends on limitations imposed by customers and/or technology. Another participant, more directly, said that he does not agree with the statement justifying that: This statement indicates that evaluating success only at the end makes a limited contribution. This makes it pertinent and important to adopt success evaluation practices from the beginning of the project. If deviations happened, there would be time to correct them.
The following statement, by one of the participants regarding the importance of evaluating success, is also noteworthy: "The post-mortem analyses from success evaluation provide valuable lessons learned for the next projects. It is also important to make a systematic control -during the project life cycle -of the different global objectives of the project: scope, time, cost, quality. Only this way, monitoring in relation to a baseline is it possible to "guarantee" that the final result will be in line with what was intended by all stakeholders (customer, organization, team, suppliers, etc.). In addition to systematic success assessment practices, other aspects must also be taken into consideration, such as good planning, good communication, good team management, and good stakeholder management."

Insights from additional interviews
The use of a combination of quantitative and qualitative methods can bring complementary insights to research (Bentahar & Cameron, 2015). To deepen on some answers obtained in the survey, related to the "how" and "why" IS project managers adopt certain criteria and success evaluation processes, additional interviews were conducted with IS project managers with different profiles.
Interviews with IS project managers have been used in research projects for different purposes such as to identify skills and competencies that drive project success (Napier et al., 2009;Skulmoski & Hartman, 2010), to identify elements that impact success in virtual work settings (Verburg et al., 2013), or even to understand the mentoring relationship for achieving success in IS projects (Leong & Tan, 2013). According to Gubrium and Holstein (2001), in an interview, the interviewer coordinates a conversation to obtain the desired information. The questions are answered in a more or less predictable format until the interviewer's agenda is completed and the interview is over.
For selecting the experts, the research team approached several project managers from their personal contacts, asking them about their experience regarding success evaluation and availability for an interview. This enabled the selection of experts with different profiles concerning the knowledge/practice on the success evaluation process, assuring richer and complementary insights.
The interviewees of this research have an academic background in IS (two of them have a Ph.D. degree and the other two have an M.Sc. degree) and all of them have extensive experience as IS project managers (over 10 years each). For anonymity purposes, they are identified as PMx: PM1 never had applied formal processes for success evaluation in his projects, and he is not aware of formal processes; PM2 is aware of the success management process, but never applied it in practice; PM3 started to use formal processes very recently; and PM4 has extensive experience in the management/evaluation of success formally.
Regarding the question related to who should participate in the definition of the evaluation process, the answers were in line with the survey's results: the project manager and the client being mentioned as the most important stakeholders. The explanations given by two of the respondents were: "The project manager and the client are involved on the verification of the project status and decision-making processes, so they must understand and define what represents the success of the project and, consequently, its evaluation process." (PM3) "The project manager needs to understand the project (concerning success) to be able to evaluate and report it. Success evaluation is also an important tool for communication, bringing formal evidence of the project's "health" to share with other project stakeholders." (PM4) In the survey, the sponsor and the project team were also cited as participants in the definition of the success evaluation process. PM4 justifies it by saying that: More than one-third of the survey's participants stated that there is no defined success evaluation process.
Interviewees were asked about what might be the reason for that: "The project management actions need to be authorized by the top management of the organization. The lack of maturity and culture of senior management prevents this type of evaluation from being included into project management practices, since the focus is almost exclusive on the budget and the final delivery of the project." (PM1) "In practice, many project managers rely on project management guides and methodologies. The assessment of success is not clear in these guides, which impacts the lack of use in organizations." (PM3) "One of the reasons may be related to a low maturity of the organizations' project management practices, not having (project management) processes clearly defined. Other reason maybe the fact that organizations are unaware of the success management role and importance, not being aware of its benefits." (PM4) About the milestones of the success evaluation process, this survey showed that, when defined, it happens mainly in the initiation and planning phases. The justifications of the interviewees were: "I believe that evaluation procedures can be established right from the business plan definition. Because, if this is defined at the beginning, it can be gradually improved throughout the project." (PM1) "The right moment to define would be in an early phase of the project (initiating and planning) because certain factors impact decisions from the beginning of the project." (PM3) This research has shown that evaluation of success usually takes place only at the project closure stage. The explanation of one of the interviewees was as follows: "The evaluation of success normally takes place in the closure phase as this is when deliveries are made, even in incremental projects. If deliverables have the requested quality and the client approved them, this is many times considered as a sign of success." (PM2) Regarding the milestones for evaluating success, the interviewees said: "I believe that there could be three fixed moments of evaluation: right after the first delivery, in the middle of the project, and in conclusion. More evaluations can be added depending on the size of the project or when there is some significant change in the project." (PM3) "The moments for evaluating success should be defined in each project and in a dynamic way. A project can, for example, define monthly evaluations and then move to weekly evaluations if it shows to be important. After some time, it can be concluded that it is no longer necessary to have weekly evaluations, and go back to the monthly or even change to bimonthly." (PM4) Regarding the most-cited criteria for success (meeting deadlines, budget, scope, project objectives, and quality of deliveries), according to the interviewees, some reasons for them are: "These criteria are the most cited for the fact that they have been used for a long time, they are the «classic» ones (the Iron Triangle criteria). Another reason is that they are the easiest to measure. They are also «sounding» for top management." (PM4) Finally, regarding ways for obtaining information to evaluate success, the most cited were meetings, reports and deliverables. Some of the reasons are: "Many stakeholders evaluate success through deliverables, considering the quality and whether it were delivered on time and within budget. This assessment takes place through reports and meetings." (PM3) "As in many organizations success evaluation is carried out without formal processes, meetings are a straightforward way of getting a perception of success." (PM4)

Conclusion
It is well accepted and somewhat consensual that IS are critical for the development of any organization (Penalver, 2018;Varajão, 2018a). Projects are a primary way of structuring the activities and resources needed for improving an organizational IS. An IS project is a temporary endeavor undertaken to create a unique product, service, or result, and can assume many forms such as the deployment of commercial-off-the-shelf applications or consultancy assignments. IS projects are socio-technical undertakings, carried out to improve organizations and achieve business benefits (Varajão, 2018a).
Despite its importance, evaluating an IS project is not an easy task (Liberato et al., 2015) and resources are needed for the evaluation -from the initiation, when the project comes up, until the post-project (when the effects of project implementation are at stake). Moreover, it is not always easy to justify to top management that the assessment of success, beyond the associated costs, is beneficial to the organization, since it provides an opportunity to think about what happened in the project and to identify opportunities for improvement. Furthermore, while some of the dimensions of success (e.g., time and cost compliance) are straightforward and relatively easy to measure, others are of great complexity and cannot be measured with conventional assessment (e.g., "how to measure the contribution of a particular project to improve the company's market image?"). As far as some criteria are concerned, a quantitative assessment is many times possible. In other cases, it is unlikely a company will be able to do so, and thus a qualitative assessment is the viable alternative (Varajão & Trigo, 2016).
The results of our web-based survey show that, on the one hand, the definition of the success process evaluation happened in more than 40% of the cases before the project execution, but in most of the cases only had the participation of the project manager (in less than onethird of the cases there was the participation of other stakeholders, as for instance, the client). Regarding the definition of criteria to evaluate success, the Iron Triangle criteria were the most cited, with the client satisfaction mentioned only in 50% of the cases. The success evidence is collected mainly through meetings and reports, typically at the end of the project. On the other hand, in almost 36% of the cases, there was no process defined for success evaluation, and in more than 43% of the cases there were no defined criteria.
Overall, results show that in many cases the process for evaluating success is not formally defined or even put into practice. Regarding the criteria to evaluate success, the Iron Triangle criteria (scope, cost, time) continue to be the most frequent. Many times, the reported success of projects is the result of perceptions and not of formal evaluations. A limited view on the success of a projectfocusing only on time, cost, and scope -may lead projects to be managed based on an incomplete set of goals and may subsequently conduct to a feeling of dissatisfaction of the different stakeholders. Despite the success being currently viewed as multidimensional, with technical, economic, behavioral, business, and strategic dimensions (Bannerman, 2008), in practice this is not evident in the formal measurement of a project's success, requiring new research efforts (Varajão & Trigo, 2016). Figure 2 shows a summary of the main insights from the study.
This study took a practice lens to explore the process for evaluating success in IS projects, mainly in what concerns to the process definition and implementation. It makes three key contributions. First, it contributes to the IS and project management body of knowledge, allowing a better understanding of what project managers do to evaluate success in their projects. It also provides explanations for the different practices found in the field as well as a fresh view on important aspects for evaluating success such as the used criteria and participating stakeholders. For instance, considering that the IS Success Model (Delone & McLean, 2003) and IS Implementation Success (Bradford & Florin, 2003) are almost 20 years old, this research shows that the variables of these models remain important for IS projects. It also presents new criteria that should be considered in IS project success management models. Second, it contributes to practice, raising attention of practitioners for fragilities in the project management daily practice regarding the success management in general and formal success evaluation in particular. Furthermore, it provides details on the success management implementation, such as when the evaluation process can be defined, which criteria can be defined, who should participate in the evaluation, when the evaluation should take place, among other aspects. Project managers can use the results to justify and improve the evaluation of success practice and, consequently, the success of their projects. Third, for education, it draws attention to the need of incorporating success management into course syllabus, since this seems to be a less explored subject in education and training.
There are some limitations of this research which are mainly related to sample size and coverage. Even though the respondent managers are experienced and working mainly in organizations with international presence, the sample could be expanded to other geographies and company sizes. The scarce literature on the topic (focusing the evaluation of success in practice) also limited the comparison of results.
For future work, data from different countries should be gathered, using a larger sample size and combining different perspectives (e.g., other stakeholders). It is also important to replicate this study in projects of other areas and industries (e.g., construction), to gain a better understanding of the phenomena. Another research avenue regards to relating the level of success of projects to the success management practices (including the formal definition and evaluation of success).

Funding
This work has been supported by FCT -Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020.

Notes on contributors
Jaime Pereira is a researcher at the Prochild CoLAB. He obtained a Master's Degree in Engineering and Management of Information Systems (2016), at the University of Minho. In the past, he has developed IT projects related to industry 4.0

EvaluaƟon
of IS projects success The reported success of projects is many Ɵmes the result of only percepƟons and not of formal evaluaƟons Figure 2. Summary of main insights from IS project success management practice. and digital health. Nowadays, he develops IT projects related to childhood poverty and well-being. His interests include: information systems (IS) project management; software engineering, dealing with topics such as Industry 4.0, Internet of Things, cloud computing, IS interoperability, logical and technical IS architectures (cloud computing, SOA, deployment architectures, microservices and monolithic architectures); agile approaches (SCRUM) for IS projects; and digital health.
João Varajão is currently professor of Information Systems (IS) and Project Management (PM) at the University of Minho. He is also a researcher of the ALGORITMI center.
His current research interests are in IS PM and IS Development (addressing, particularly, IS and PM success). Before joining academia, he worked as an IS consultant, project manager, IS analyst, and software developer, for private companies and public institutions. He has published numerous refereed publications, authored books, edited books, as well as book chapters and communications at conferences. He serves as editor-in-chief, associate editor, and member of committees for conferences and international journals.
Nilton Takagi is a professor at the Computing Institute of the Federal University of Mato Grosso (UFMT). He is also a researcher at the ALGORITMI research center and PhD student in Technologies and Information Systems at University of Minho. He holds a Master in Informatics from the Federal University of Paraiba and MBA in Project Management from the Getulio Vargas Foundation, is a specialist in systems analysis from the Pontifical Catholic University-Campinas, and holds a degree in Computer Science from the UFMT. He has experience in IT and management, working on the following topics: agile methods, business intelligence, project management, business process management, data analysis, combinatorial optimization, and metaheuristics.
is no formal definition of criteria for success assessment", "Other" (open)) Who participates in the definition of criteria for evaluating the success of projects in the company? ("Project Manager", "Executing team", "Project management team", "PMO (Project Management Office)", "Client", "Sponsor", "External consultants", "External auditors", "There are no participants", "Other" (open)) What are the ways/tools used to obtain information to evaluate the success of projects in the company? ("Interviews", "Surveys", "Reports", "Meetings", "Deliverables", "There are not used any means for evaluating the success formally", "Other" (open)) When is the success of projects evaluated in the company? ("Project initiation", "Project planning", "Project execution, one time", "Project execution, several times", "Project closing", "The project success is not evaluated formally", "Other" (open)) In your opinion, which practices for evaluating the success of projects may be most advantageous to put in practice in your company? (open)

Success of IS projects in the company (each participant reported up to three projects):
Type of project ("Software development", "Package implementation", "Systems enhancement", "Consultancy and business analysis assignments", "Systems migration", "Systems integration", "Infrastructure implementation", "Outsourcing e insourcing", "Disaster recovery", "Other" (open)) Project budget (in Euro) Project duration (in months) Project success (Likert scale 0-10, "0" meaning "Abandonment of the project" and "10" meaning "Total success") Observations (open) Part of the success or failure that sometimes happens in IS projects may be due to the lack of systematic success assessment practices. Do you agree with this statement? (open)