Agile software development project evaluation using the partial least squares – structural equation modelling ( PLS – SEM ) approach in view of critical success indicators

In this s tudy, the Agile software development process is analysed by means of success and failure cri teria, and their effects are determined using the partial least squares – s tructural equation modelling methodology. The s tudy identi fies criticial success factors in Agile software development methodology and speci fically focuses on indicators to conclude their signi ficance of relationship and impact, so that the possible results are determined, predicted and exterminated in advance. The literature search determined the success indicators of agile projects in a mul ti -dimensional view of factors. Each factor was classified into sub-factors and indicators which helped to obtain a multi-dimensional view of the factors that made them more viable. The answers of the participants were mapped to the detailed cri teria and a pplied to the model developed. The results which showed the effects of each sub-cri teria mapped to one of the main cri teria of the Agile software development process were determined and evaluated.


Introduction
Agile software development started to become popular in the late 1990s, and is now one of the most preferred software development methodologies used by organisations, project managers and developers.The Agile methodology was proposed inside of the 'Agile Manifesto' (Beck et al., 2001), which has principles by means of individuals and interactions over processes and too ls, working software over comprehensive documentation, customer collaboration over contract negotiation and responding to change over following a plan.
Agility means the power of moving quickly and easily.Larman (2004) states that agile software development method is different from the traditional, plan-based approaches (such as Waterfall or sequential methodologies) in software engineering.It aims at a fast, light, effective and qualified development life cycle that supports customers' involvement as much as possible with simple phases and quick turnarounds.In software development, applying agile methodologies means using the power of flexibility to move quickly and adaptively for applying changes over time.The main power of agile software development method is to provide a solution in increments, which starts with deployable units and is developed over time into products with fully functional, scalable units.This is why agile methodology is defined as an iterative method to make software development i n shorter times with lightweight deliverables and cycles.
Agile software development teams aim to deliver a working application at every sprint and demonstrate it to customers or related people at the end of each sprint.The relationship and communication between team members are more important than using a development tool and a pre-defined process.According to the agile development process, team spirit drives a project to success.The Agile method advocates that the relationship between team and customer is as important as the relationship between team members.Instead of deep documentation, agile software development prefers the working code that is tested periodically.When the team member merges their own code into pre-merged code blocks, agile requests that the test procedures should start automatically, and thus, successful code blocks would be ready.The most important characteristic of the agile software development is that the customer can change requirements anywhere as the project moves along.Besides, the customer should participate on the project in every phase.As a result, developers and customers should work together by giving feedback during the project phase.
The critical success factor (CSF) approach to determine and evaluate an organisati on's performance was first introduced by Rockart (1979) and then was established by Rockart and Crescenzi (1984).CSF is applied agile projects in order to define the performance criteria of an organisation and identify the measurement methods.CSFs specify the number of areas that will help to get competitive efficiency and effectiveness metrics for the team member, the team or organisation.Bullen and Rokhart [4] summarise CSFs as exact answers of what parameters take away a project from success.CSFs in software projects are determined by using experience gained from previous projects.Mansor et al. (2014) state CSFs in the software development business to be related to software engineering as well as a combination of business and project management methodologies.

Literature Review
Briefly, the thesis study reviews three main concepts: agile method and CSFs, case studies and partial least squares -structural equation modelling (PLS-SEM) modelling for evaluation.So the literature search has been focused on these three main concepts.
For the literature-based study on agile method and CSFs, Doherty (2012) used the method of getting opinions from experienced program owners and project managers to determine the contribution, explore the management approach and evaluate the success factors to the project's success.A total of 519 samples were collected from project owners who work on projects and have experience in leading IT projects.The two-phased research approach was applied to the samples by Demirel, S., Caliskan, H., Karahoca, D. & Karahoca, A. (2017).Agile software development project evaluation by using the partial least squares-structural equation modeling (PLS-SEM) approach in the view of critical success indicators.Global Journal of Information Technology: Emerging Technologies. 7(3),[99][100][101][102][103][104][105][106][107][108][109][110][111][112][113][114][115] 101 employing a frequency analysis of the preferences applied to Q-analysis method to combine and analyse the list of success factors.Then detailed evaluation was provided as an explanation for the CSFs.Nasir and Sahibuddin (2011) prepared a comparative study and used the survey methodology in the literature to determine the success factors that can potentially impact the project success.Since 1990-2010, 43 articles have been used and evaluated to propose the CSFs that affect the agile project's success.The preferred method for the study was content and frequency analysis methodologies.As a result of the analysis, 26 factors were determined as relative to the project success.Among them, the top five success factors are suggested to be carefully focused by project managers or program owners as the frequency of occurrences are more than 50% for each.Wan and Wang (2013) focused to determine the key success factors among the CSFs for agile projects.They highlight that most critical factors are dependent on the view of the project manager, who should analyse the return on investment and determine the most CSFs depending on the project and implement them.Charette (2005) determined the failure factors of an agile project, which is the opposite of those factors and are evaluated as success factors.Cockburn and Highsmith (2001) focused on the people factors specifically and evaluated the effects of the people factor and if it can lead to the success in software development projects.Chow and Cao (2008) discuss failure factors in four dimensions, namely: organisational, people, process, and technical (Tanner & Willingh 2014).Vijayasarathy and Turk (2008) designate that lack of training, unfamiliarity with agile approaches, lack of managerial support and interest, resistance from individuals, teams or organisation itself are considered to be some of the factors that lead agile projects to fail.
For case study and survey-based studies, we explored many articles and found some countryspecific studies based on surveys or questionnaires.Abdulaziz and Mayhew (2013) performed a case study in Saudi Arabia to present the success factors that can affect software projects.The study performed a two-phased method which combines quantitative and qualitative methods.In the first phase, in order to collect the data and analyse, an interview was performed.After the interview, 17 factors were proposed as the success factors.In the second phase, a questionnaire was used to evaluate and validate the proposal as a quantitative method.Wan et al. (2013) focused on the manifesto and 12 agile principles, and performed a case study of J Group by applying an adaptive model.The study determined the success factors as: 1) Build the scrum as a self-managing group and a learning organisation 2) Professional release and development capability 3) Explicit project management.
The study focused on the methodology of Scrum as J Group practices it.Ofori (2013] performed another study in Ghana.It collected the dataset by performing a survey on Ghanaian organisations.Knowledge creation theory was used in the analysis of the dataset and provided the CSFs that contribute to the survey.Nasir and Sahibuddin (2011) performed a Delphi study (five rounds) on the team software process (TSP) that aimed to determine the adherence of CSFs for agile software projects.Three experts participated in the study.The study findings supported the practices to address the best 14 success factors.The participants were agreed on the outcomes of TSP, which reproduce a very good level f or four of the success indicators, 'good' level for six of the success indicators, 'limited' level for only one of the indicators and none at the 'fair' degree.Chow and Cao (2008) used the quantitative method to gather data via an online survey, which was formed of demographic data collection and 7-point Likert scale questions.The target audience was members of Agile Alliance.First, five members of the target population tested and validated the content and provided their feedback to enhance the survey, and then the survey was spread to 83 group coordinators of Agile Alliance user groups and 60 contact people of corporate members of the agile.The survey period lasted for 6 weeks and a total of 408 people responded and 109 projects were submitted with comprehensive data.
Another example is that of Stankovic et al. (2013), who collected the data in the study by using an online survey in the form of a 7-point Likert scale.The survey was spread to the target audience consisting of managers, developers and experts in former Yugoslavia IT companies.There existed four sections in the survey, including demographic or personal data, success factors, insights of success, additional notes and feedback.After a 1-month survey period, 23 complete responses were collected.
For PLS-SEM, the following articles were studied and investigated: Campanelli (2016) searched for the impacts of tailoring criteria that can be used on adoption of agile software development methodologies.His study first focused the tailoring criteri a available based on the literature search.Then, a model for agile practices adoption was proposed with the base of the tailoring criteria.Survey was used to collect the data among agile professionals and PLS-SEM was used to evaluate the model proposed on the dataset.The literature search showed that agile methods tailoring is an active research theme, the fundamental tailoring approaches are not specific to an agile method, the majority of the research used empirical research procedures, and that tailoring is mainly developed by using systematic method engineering approaches.The model has been validated and presents the effect of the external and internal environment with previous knowledge and experience tailoring criteria on agile adoption.They also highlight organisations' select agile practices according to their needs and tend to use custom methods or hybrid software practices.The proposed model can help the selection of agile methodologies, based on the level of importance each of the tailoring criteria has on the organisation's context for adoption.
Senapathi and Srinivasan (2014) published a study to validate and test a continuing agile usage or post-adoption based on a survey study.Survey data was validated using PLS-SEM models with variance and structural equations implemented in SmartPLS 2.0.Reliability was checked with a special focus on developing valid measures.
It is observed with the literature search that success criteria and researches are mainly based on either the case studies, personal observations of the experts from different agile practices or regression techniques applied to the data that gathered with different questionnaires or surveys.
Based on the literature search, Table 1 depicts the success factors and indicators to be an alysed in five dimensions: organisational, people, process, technical and project (Chow & Cao, 2008).Table 2 shows the failure factors in four dimensions.Demirel, S., Caliskan, H., Karahoca, D. & Karahoca, A. (2017).Agile software development project evaluation by using the partial least squares-structural equation modeling (PLS-SEM) approach in the view of critical success indicators.Global Journal of Information Technology: Emerging Technologies.7(3), 99-115. 103

Data Collection
In this section, we describe our dataset and the methods that we used to test our dataset.As we observed in the literature search, we also refer to a case study to collect the data and build up the proper dataset by using the survey method.Data were gathered with the use of an online survey that was spread to the target audience consisting of executives, managers, developers and customers in Turkey IT companies.Demirel, S., Caliskan, H., Karahoca, D. & Karahoca, A. (2017).Agile software development project evaluation by using the partial least squares-structural equation modeling (PLS-SEM) approach in the view of critical success indicators.Global Journal of Information Technology: Emerging Technologies.7(3), 99-115.105 The survey period lasted 2 months and a total of 172 people (124 male and 48 female) responded to the online survey.The average years of experience of the respondents in software development was 6.4 years and the average years of agile experience was 3.3 years.The average number of agile projects involved in by the respondents was 9.6.Demirel, S., Caliskan, H., Karahoca, D. & Karahoca, A. (2017).Agile software development project evaluation by using the partial least squares-structural equation modeling (PLS-SEM) approach in the view of critical success indicators.Global Journal of Information Technology: Emerging Technologies.7(3), 99-115.

106
The 5-point Likert scale was chosen in the survey to be formed with the statements: Strongly Disagree, Disagree, Neither agree or disagree, Agree, Strongly Agree and they were codified by assigning five to the highest statement 'Strongly Agree' and assigning one to the lowest statement 'Strongly Disagree'.
Then, the questions were associated and mapped to the main factors and to the sub-factors in each of them.For instance, question 17 aimed to figure out people's idea and feedback on the impact of absence of management support under organisation dimension; similarly, questions 13, 28 and 52 were linked to the insufficient experience under people dimension.In the case of multiple questions logically associated with the same main factor and the same sub-factor, in order to reflect all answers for more accuracy, the average (arithmetic mean) of all these answers was evaluated and assigned to the indicator (e.g., 4.3 points).
The same evaluation was applied to all the questions in the survey (Section 1.2 Personal Influencers and Section 2 Agile Development Methodology Failure Factors, for all respondents (172) in the survey).
The data collected from the survey were analysed using PLS-SEM or PLS Path Modelling (PLS-PM), based on the literature review.It is a statistical method for modelling complex relationships (structural equation models) among latent variables (LVs) and manifest variables (MVs) (observed variables).
PLS-SEM or PLS-PM was also used to display the model in a graphical format, using what is called a path diagram that represents in a visual way the relationships stated in the model (Sanchez, 2013).

Methods
We have focused on PLS methods for determining the existence and impacts of success factors in the agile project, which are PLS-SEM and PLS-PM methods.Hair et al. (1988) stated that these two methods can be used to model the complexity of causeeffect relationships among the LVs.Vinzi, Chin, Henseler & Wang (2010) highlight PLS-PM aims to increase the number of variances rather than accuracy of the statistical estimates, so it does not provide a covariance matrix.
The description of the modelling is based on two models: the outer model (also called the measurement model) and the inner model (also called the structural model).The outer model measures the correlation of the MVs to their LVs and the inner model endogenous LVs to other LVs.Lee, Petter, Fayard and Robinson (2011) describe the algorithm that provides the structural equation model and determines the estimates of LVs in alternating steps by using the inner and outer models.The outer mode performs calculations on LVs using the weighted sum of its MVs.The inner model performs calculations on LVs using the linear regression between LVs and MVs.These calculations are performed repeatedly until proper convergence results are received.Peng and Lai (2012) make the definition of an LV as a construct (an unobservable, indirect variable) which is constructed with observable, measurable, direct variables, formulised as xh, which are the indicators or MVs.Sarstedt et al. (2014) describe the ways of determining the LVs with their MVs, which are indicated with three methods: reflective way, formative way and the multiple effect indicators for multiple causes way.In this study, the reflective way has been used for analysis.

Findings
In the first step, the initial model was built and reliability analysis was performed on the success and failure criteria.Then, depending on the reliability results, models were reconstructed to build the final models.In the second step, validity analysis was tested on the final models and the results were evaluated.

Findings of the Success Criteria
The initial PLS-SEM diagram was evaluated on the LVs and associated MVs from the real life survey to measure the impact and factors that led the agile projects to success.There are five main CSFs as shown as LVs in the initial model: Organisation, People, Process, Technical and Projects as shown in Figure 2 and an exit factor has been determined as 'Success Factor' and also shown as LV.For each LV, the sub-criteria of the factors are added as MVs to the LVs.After executing the initial model, two-step analyses was performed on the findings.First, the outer model and then the inner model were validated.Demirel, S., Caliskan, H., Karahoca, D. & Karahoca, A. (2017).Agile software development project evaluation by using the partial least squares-structural equation modeling (PLS-SEM) approach in the view of critical success indicators.Global Journal of Information Technology: Emerging Technologies.7(3), 99-115.Bagozzi and Yi [3] define, for Cronbach's Alpha, 0.4 or higher for explanatory research and 0, 70 or higher for factor reliability, and for D.G. Rho, 0, 60 or higher for explanatory research and 0, 70 or higher for composite reliability.
In the initial model, Cronbach's Alpha and D.G Rho cannot be computed, so validity check was performed with cross loadings.Cross-loadings are used to determine the effectiveness of each factor on the other factors (non-target).It is one of the methods used to decide whether MVs are effective enough on the LVs and further analysis can be performed on the model or not.If constructs are valid, there should be high correlations (>0.5) between cross loadings of the same construct.If constructs are not valid, they can be removed from the model to construct a better model with high validity.This operation is performed repeatedly with each result until improvement is noticed on the construct validity of the indicators.
The latest PLS-PM model is shown in Figure 3. Based on the fact that if constructs are valid, there should be high correlations (>0.5) between cross-loadings of the same construct, which in the final model cross-loadings of the MVs are all higher than 0.5.
According to the Composite Reliability indexes (explained in Table 4.2), Dillon-Goldstein's rho results were greater than 0.7 and the first eigenvalues of each LV were bigger than the others, thus each and every LV block consisted of MVs is verified to be unidimensional.In other words, the reliability values of this model were satisfactory and moderately affected the model.After checking the reliability of the model, it was considered as the 'final model' in the remaining parts of this study.Final model is confirmed reliability analysis with cross-loadings first and now further analysis can be performed on the model.

The Inner Model (Structural Model) of the Success Criteria
For validation of the structural model, R² measures and path coefficient values were used.As PLS-SEM method tries to determine the relations of the endogenous LVs and prediction-oriented approach is used for building the models, the R² values are expected to be high enough to meet the purpose.The expected values for R² depend on the discipline of the research.To determine the success drivers, R² > 0.75 is evaluated as high, whereas 0.20 may be evaluated as high in determining the consumer behaviours.This study focuses on determining the success indicators of agile projects; 0.75 is used as the reference value in R² squares.
Another validation in structural model is using the goodness-of-fit (GoF Index) value which measures the relativity among variance and covariance from the sample matrix.GoF Index measures the relativity and is one of the ways to determine the model fit.state that the GoF Index should be 0-1, where the values closest to 1 are considered as good model fits.In the final model, the absolute GoF index is 0.493 with the relative GoF as 0.850 (close to 1).
Table 6 illustrates the model assessment.Organisation, process and project are e valuated as the exogenous factors, whereas technical, people and success are the endogenous factors.As success factor was used as the exit criteria, it should be endogenous, which fits with the model.In the first PLS-SEM diagram in which all LVs were connected to Failure LV, the execution of the model did not produce satisfactory results to continue with the model (GoF value was 0.302, R 2 value of the failure LV was 0.502, AVE values of all LVs were lower than 0.5).4.2), Dillon-Goldstein's rho results were greater than 0.7 and the first eigenvalues of each LV were bigger than the others, thus each and every LV block consisting of MVs is verified to be unidimensional.In other words, the reliability values of this model were satisfactory and moderately affected the model.If the Cronbach's alpha values were examined, the most effective variable (the highest score) was found to be the organisational variable, which was 0.805.Demirel, S., Caliskan, H., Karahoca, D. & Karahoca, A. (2017).Agile software development project evaluation by using the partial least squares-structural equation modeling (PLS-SEM) approach in the view of critical success indicators.Global Journal of Information Technology: Emerging Technologies.7(3), 99-115.113

The Inner Model (Structural Model) of the Failure Criteria
Table 9 demonstrates the results of the structural model estimates for the model using all data from the survey.R² which was defined as a coefficient of determination was 0.737 and could be considered to be substantial (Hair et al. 2011).
Table 10.R 2 of failure Also, the absolute GoF index value was calculated as 0.452, which was an acceptable value in a real case model.The relative GoF index value was evaluated as 0.862, which could be considered very high.

Conclusion
This paper was an attempt to evaluate the impacts of CSFs for agile software development projects and specify the success and failure criteria based on regression methods applied to a proper dataset.
A proposal framework is presented by modelling the multi-dimensional view of the success factors, based on the five categories (people, project, organisation, process and technical) with their main and sub-indicators.The failure factors and indicators were examined in four dimensions (organisational, people, technical, process) and their sub-categories that mainly contributed to the software development methodologies and to the agile specifically.Multi-dimensional view narrows down the model and increases readability and applicability.This research was based on the online survey data to explore the critical factors of agile software development projects using quantitative approach.PLS-SEM was effectively chosen to construct a model and analyse the data to determine the factors and indicators and their relative (weighted) impact on the agile projects.
In the successful model, based on the responses of the survey, the success factor was related to three main factors: people, project and technical.Technical factors are evaluated as having relatively high association on success and success factors are defined at 76% on two sub -criteria of delivering the project on time with quality.Technical factor includes both technology properties and infrastructure.Technology determines the development environment with coding standards that will be followed, and infrastructure includes technical trainings, integration testing, automation, Demirel, S., Caliskan, H., Karahoca, D. & Karahoca, A. (2017).Agile software development project evaluation by using the partial least squares-structural equation modeling (PLS-SEM) approach in the view of critical success indicators.Global Journal of Information Technology: Emerging Technologies. 7(3),[99][100][101][102][103][104][105][106][107][108][109][110][111][112][113][114][115] documentation and regular delivery to the customer.All these sub-items in technical factor are related with the success of the project.The people factor is also evaluated as one of the factors as in agile projects team synergy, efficiency and output are critical to perform continuous delivery to the customer.People factor includes not only the technical skillset and expertise but also communication skills within the team or with the customer.Project factor determines technology based on the content, and defines the project type such as an integrati on project or development projects.Project factor has potential impact on the success as depending of the project type, size and nature, the success factor may be challenged.The percentage of the success ratio is evaluated as low when the project has a variable scope in its requirements; especially, it has emergent requirements and requires multiple dependent teams such as distributed international projects.
Based on the failure model that was developed and analysed, the technical factors and indicators (e.g., no or long delivery cycles, lack of developer involvement in prioritisation, etc.) were revealed to dominantly lead agile projects to fail.The process factors and indicators had unexpectedly lower and negative impact on agile project failures, though, if the role of the customer was vague, it was seen as a factor to cause agile project failures.Similarly, resistance from teams or individuals (people factor) and traditional/outdated culture and unsuitable environment (organisational factor) were also determined to lead agile projects to fail considerably.Technical factors itself were internally affected by process factors mostly (higher than 95%) and by people factors to some extent (almost 3%).
Based on Chow and Cao (2008), agile success and failure factor research, incorrect delivery strategy, improper agile software engineering techniques and absence of a high-calibre team were found to be critical failure factors, leading the agile project to fail.This research similarly indicated that the technical factors and no or long delivery cycles were obviously impacting agile projects negatively.
This study provides an empirical model and can be improved with further analysis.Although there were further studies around CSFs mostly in agile development, failure factors or indicators were not observed so much to be focused specifically.With the help of this study, critical factors and indicators were primarily studied and should be considered as an example or reference study for further research.

Figure 2 .
Figure 2. Initial PLS-PM success model 4.1.1.The Outer Model (Measurement Model) of the Success Criteria Composite reliability (monofactorial MVs) shows the internal consistency by using cross -loadings, Cronbach's Alpha (1971) and D.G. Rho values.Cronbach's alpha takes into account the equal weighting of the indicators, whereas the empirical model, D.G Rho, assumes indicators are unequally weighted.Bagozzi and Yi [3]  define, for Cronbach's Alpha, 0.4 or higher for explanatory research and 0, 70 or higher for factor reliability, and for D.G. Rho, 0, 60 or higher for explanatory research and 0, 70 or higher for composite reliability.
Figure 3. Final PLS-PM success model

Figure 4
Figure 4 depicts the first result of the initial PLS-SEM diagram evaluated on the LVs and associated MVs from the real-life survey to measure the impact and factors that lead the agile projects to fail.

Figure 4 .
Figure 4. Initial PLS-SEM diagram on agile failure factors

Table 2 .
Failure factors and indicators in agile

Table 3 .
Survey questions, corresponding references and covered items

Table 5 .
Composite reliability of final model

Table 8 .
Composite reliability table

Table 9 .
The result of structural model assessment and R² values