Why Most IT Projects Fail. And How Agile Principles Help

by Kelly Waters

why most IT projects fail, and how agile principles help

I recently wrote a blog post explaining why most IT projects fail to meet expectations.

As a follow-up, here I take a quick look at the common reasons for project failure, and how I think agile software development methods and agile principles mitigate these risks and issues.

I have a strong view that agile methods help significantly with a lot of these areas of project risk, although I'm sure not all. This is an enourmous subject, so I can't really do it justice and keep this blog post a reasonable length. So you'll just have to take or leave my comments on face value :-)

Here's how I think agile principles help, in my experience:

Common cause of failure	How agile helps
Project Initiation & Planning Issues
Unclear or unconvincing business case	Agile principles don't help directly with the busines case - in fact it can be hard to make a business case for agile projects due to their agile (i.e. unpredictable) nature. However the principles of incremental delivery and frequent deliver of products can help to get an initial solution out, test the proposition with the market and get real customer feedback to inform the priorities for further development.
Insufficient or non-existent approval process	Agile business cases and new business propositions benefit from the same diligence and challenge before investing large amounts of money, just as in any other approach to development.
Poor definition of project scope and objectives	Agile projects also benefit from clear definition of scope and objectives, even though details are allowed to emerge throughout the development.
Insufficient time or money given to project	If only agile could solve this!
Lack of business ownership and accountability	Active user involvement, or involvement from a key user representative in the business, creates an environment that fosters close collaboration and cooperation.
Insufficient and/or over-optimistic planning	This is an interesting one. I (like many agile enthusiasts) believe that it's practically impossible to plan every detail of many software development projects up front, hence expectations are better managed by active involvement in the project, frequent delivery of product on an incremental basis.
Poor estimating	Agile methods provide some important principles to help with accuracy of estimating: estimating should be done by the whole team as a collaborative process; tasks should be broken down into micro pieces (ideally less than 1 day) so progress is measurable on a daily basis; 'velocity' is calculated on the number of estimated hours delivered only when a feature is 100% complete, providing a gauge for how much development can be safely included in future iterations. In non agile methods, this approach can also be adopted and is known as 'earned value analysis'. So even if you're terrible at estimating, this approach can be self correcting as long as you're consistently terrible :-)
Long or unrealistic timescales; forcing project end dates despite best estimates	Agile projects encourage short and regular iterations, developing the software and delivering working product in small bitesize pieces.
Lack of thoroughness and diligence in the project startup phases	Rather than diligent and thorough planning, agile principles propose to deliver small increments of working product and get continuous feedback from active user involvement throughout the development cycle.
Technical & Requirements Issues
Lack of user involvement (resulting in expectation issues)	Active user involvement and continuous feedback is one of the most important principles of an agile approach.
Product owner unclear or consistently not available	One of the reasons product owners are unclear in traditional projects is because they are asked for far more detail than they can handle, too early in a project and when they cannot visualise the solution. Instead, agile requirements are kept lightweight and visual, and delivered just in time for a feature to be developed. Availability must be forthcoming for agile principles to work, as it's essential for constant collaboration.
Scope creep; lack of adequate change control	Agile projects may stick to the broad scope of the project, but requirements are allowed to emerge and evolve. However the project must include non-essential requirements at the outset, in order for emerging requirements to be traded with original scope.
Poor or no requirements definition; incomplete or changing requirements	Agile projects expect requirements to be incomplete and changing. That's the nature of software. Instead of resisting this, agile projects provide for it by allowing requirements are allowed to emerge and evolve. Requirements being produced on a feature-by-feature basis, just in time to be developed, helps with definition because it breaks this intensive task into small pieces instead of being a mammoth effort up front.
Wrong or inappropriate technology choices	Agile projects can surface inappropriate technology choices early, as they encourage frequent delivery of product on an incremental basis. Testing is integrated throughout the development cycle, testing each feature as it's developed. Doing so can help to ensure inappropriate technology choices are identified early, before too much of the software has been developed.
Unfamiliar or changing technologies; lack of required technical skills	Agile methods don't help directly with this issue, although can help to surface such issues early, and make them visible.
Integration problems during implementation	Agile projects are delivered in short iterations, with testing integrated throughout the development. This requires continuous integration of the code and frequent builds, removing the need for a lengthy or problematic integration phase at the end of the project.
Poor or insufficient testing before go-live	Testing is integrated throughout the development.
Lack of QA for key deliverables	Working software is the key measure of progress, as the software is developed and delivered in regular iterations. This helps to ensure that the adequacy of any other deliverables is highlighted early and made visible.
Long and unpredictable bug fixing phase at end of project	Testing is integrated throughout the development.
Stakeholder Management & Team Issues
Insufficient attention to stakeholders and their needs; failure to manage expectations	Active user involvement ensures two way feedback throughout the development.
Lack of senior management/executive support; project sponsors not 100% committed to the objectives; lack understanding of the project and not actively involved	All projects need this, agile or otherwise.
Inadequate visibility of project status	Agile projects provide clear visibility of measurable progress on a daily basis.
Denial adopted in preference to hard truths	Humans, eh? Who needs 'em!
People not dedicated to project; trying to balance too many different priorities	I don't think this ideal is specific only to agile methods, but agile principles propose small, multi-displined teams dedicated to the development of the product.
Project team members lack experience and do not have the required skills	Agile principles may help to surface such issues early, as they may well be evident in early iterations of the software. Frequent delivery of iterations and continuous testing can help to mitigate this risk when it might otherwise go unnoticed until much later in the project.
Team lacks authority or decision making ability	Agile teams must be empowered.
Poor collaboration, communication and teamwork	Close cooperation and collaboration between all stakeholders is essential.
Project Management Issues
No project management best practices	Can be an issue when applying agile methods, unless using an agile management practice such as Scrum.
Weak ongoing management; inadequately trained or inexperienced project managers	Agile methods and principles are just management tools. A fool with a tool is still a fool!
Inadequate tracking and reporting; not reviewing progress regularly or diligently enough	Agile practices have daily status reporting built into the process, providing clear visibility and measurable progress on a very regular basis.
Ineffective time and cost management	Daily visibility of measurable progress.
Lack of leadership and/or communication skills	Sadly, adopting agile principles doesn't make inspirational leaders. However agile principles do encourage a particular form of servant leadership, empowering the team to take responsibility and make timely decisions.

Of course most of the things I've cited as agile mitigations can be applied in any project methodology, including waterfall. It's just that they are more the norm and explicitly emphasised in agile principles and practices, which is something I really like.

Most of all, agile principles help enormously with visibility, collaboration and engagement. This can transform the levels of trust and teamwork between the product team and its key stakeholders. The consequence of this is enhanced satisfaction, due to a much greater understanding of the commitment and expertise of the team, and the issues and risks they face. If failure, by definition, is a problem of not meeting expectations, you're half way there when you adopt an agile approach.

However, all this, in the end, ultimately still relies heavily on the skills of the team. Just because agile methods state these principles and have some inherent risk management built within the process, it doesn't mean the team necessarily has all the skills and experience it needs to execute the princples consistently.

See also:
Most IT Projects fail. Will yours?
DIY Guide for Fixing a Failing Project - by Mike Cohn
10 Key Principles of Agile Software Development
10 Good Reasons to do Agile Software Development