Everything your competitor knows about Risk Identification on Software projects

By spending the time to follow the steps in this article you should be able to identify over 80% of the risks on your project.

Risk = Probability * Impact

All projects have risks. Proactively addressing those risks will save you much needed time and energy when you need it most. This is a guide detailing the most useful methods of identifying risks on your software project, regardless of whether you are guided by a Waterfall approach, Agile, Six Sigma, PMI or PRINCE2.

Of note: Common risk areas

Most of your project risks are likely to arise out of the following prioritised areas (according to a study from Tharwon Arnuphaptrairong):

1. Misunderstanding of the requirements
2. Lack of management commitment and support
3. Lack of adequate user involvement
4. Failure to gain user commitment
5. Failure to manage end user expectation
6. Changes to requirements
7. Lack of an effective project management methodology

This study by Kloppenborg, T. J. & Tesch, D. (2004) lists 70 such risks that you can cross reference, with 745 assigned strategies concerned with risks categorised by Process (56%) and People (44%)

Risk Identification Techniques

Interview

Initial one on one interviews (formal or informal) always highlight project insights beyond the common shared knowledge that should be documented in the Business Case / Project Charter documents. Come prepared to these interviews with leading structured questions and take note of not just the risks, but also of any business drivers, assumptions and constraints (for use in CAA process (below)).

Questionnaire

This basic questionnaire (full questionnaire) below follows Boehm Top 10 Software risks to identify risks and work through suggested mitigation strategies. While I find this questionnaire the most useful there are others of note from Wallace & Keil, an international study by Schmidt et. al and one from Han & Huang all of which have been nicely documented in a research paper by Tharwon Arnuphaptrairong.

1. Personnel shortfalls

Question: Are the required number of skilled skills available for the project time frame?

Mitigation: Staffing with top talent, job matching, team building, key personnel agreements, cross training

2. Unrealistic schedules and budgets

Question: Were all requirements & tasks estimated by the team undertaking the work?

Question: Have all dependencies between tasks been identified and accounted for?

Mitigation: Detailed milestone cost and schedule estimation, design to cost, incremental development, software reuse, requirements scrubbing

3. Developing the wrong functions and properties

Question: Are the requirements for the system clear and understandable and stable?

Question: Is there a history of misinterpretation or miscommunication between developers and those defining product requirements?

Mitigation: Organizational analysis, mission analysis, operations-concept formulation, user surveys and user participation, prototyping, early users’ manuals

4. Developing the wrong user interface

Question: Has the user interface been prototyped and tested on sample user groups?

Question: Are user interface goals defined, documented?

Mitigation: Prototyping, scenarios, task analysis, user participation

5. Gold plating

Question: Is there a history of unneeded functionality being added during project development?

Mitigation: Focus on MVP, Requirements scrubbing, prototyping, cost-benefit analysis, designing to cost

6. Continuing stream of requirements changes

Question: Has past history with customer shown that requirements change often?

Question: Does the team have a good understanding of what the customer/user wants in the system?

Mitigation: High change threshold information hiding, incremental development (deferring changes to later increments)

7. Shortfalls in externally furnished components

Question: Are there any delivery risks identified with projects to which this one is dependent?

Mitigation: Benchmarking, inspections, reference checking, compatibility analysis

8. Shortfalls in externally performed tasks

Question: Is this project or any of its tasks dependent upon the completion of external project development efforts?

Mitigation: Reference checking, pre-award audits, award-fee contracts, competitive design or prototyping, team building

9. Real-time performance shortfalls

Question: Have requirements for system response time been defined?

Question: Have requirements for data storage, memory & processing speed been defined?

Question: Have user support (number of) requirements been defined?

Mitigation: Simulation, benchmarking, modeling, prototyping, instrumentation, tuning

10. Straining computer-science capabilities

Question: Has the organization delivered a similar system using similar tools?

Mitigation: Technical analysis, cost-benefit analysis, prototyping, reference checking

Brainstorming

Gather the various stakeholders into a room to try to generate as many risks as possible. There are no bad suggestions. Initially, you are looking for as many risks to be mentioned as possible (divergent thinking) before focusing in on some risks and building on them (convergent thinking). If the group get stuck, then steer them towards the various project objectives (schedule, scope, quality, budget) and tasks (requirements, testing etc.)

You may want to use this link to plan your approach to your Brainstorming session and utilise these best practices.

Constraints and Assumptions Analysis (CAA)

Every project has known assumptions, documented or not. Every time a requirement, function, task or User Story is sized it is built upon underlying assumptions. In order to analyse the work to be undertaken you should note your dependencies and business and technical constraints. The exercise to turn these into risks is straight forward. Simply state that each of these assumptions and constraints are false. This is the Risk. Now describe the consequence. This is the Impact. Whether to deal with the impact or mitigate the risk in the first place will be dealt with in a later article.

An example could be:

Assumption: The ACME payment solution provider will be used to take payment on the site.

Inverted Assumption: The ACME payment solution provider will not be used to take payment on the site.

Why would this have happened? Examples could be that the contract will not be signed in time, or your market currency is not supported. Maybe there are legal issues preventing deployment or ACME’s rates are too high.

There are many potential issues (which could be brainstormed) that could arise. Each one is a potential risk. Identifying them early allows you to take the necessary steps to investigate their likelihood and/or take steps to mitigate them.

Standard deviation from three point estimates

Some practices are just learned through experience. This one is mine. I identify so many potential risks based on this method. There is complex logic behind it, but can simply be explained as this: In three point (PERT) estimation, if the difference between the smallest number (optimistic) and the largest number (pessimistic) is large, then you have uncovered a risk. This usually happens without the person estimating the task even realising it.

No Risk identified

• Optimistic estimate = 3
• Most Likely Estimate = 5 (+2)
• Pessimistic estimate: = 7 (+4, +2)

Risk identified

• Optimistic estimate = 3
• Most Likely Estimate = 5 (+2)
• Pessimistic estimate: = 12 (+9, +7)

The PERT formula is: (O+P+4×M)/6, in Excel the PERT formula is:

=ROUND(((A1+(B1*4)+C1)/6),0)

Conditional Formatting screenshot

The Standard Deviation (SD) or σ is σ = (P — O)/6, in Excel I use this formula coupled with Conditional Formatting to ‘raise a red flag’ if SD is greater than ‘1’:

=IF((C1-A1)/6<1,””,((C1-A1)/6))

Force Field Analysis (FAA)

Your project already has known objectives and driving forces. Weaken the focuses behind them, identify why they may have weakened and identify the impact on the project. How can you mitigate against this? A much more detailed overview of FAA is found here.

Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis

Identify the strengths and weaknesses of the end-product in the market and translate these all to risks. If you are not convinced to undertake the academic analysis, here are some compelling reasons why you should.

Fishbone/ Ishikawa Diagram (Cause and Effect)

Diagramming is usually used during Risk Analysis, but introducing this during the Risk Identify stage can help identify root causes as well as various factors leading to the risk. For this reasons the Fishbone Diagram (Cause and Effect) earns its great reputation as a method to determine the root cause of the problem. This can be useful when converging on a risk. Once the root cause has been identified, the risk can be mitigated.

Cyclomatic complexity

The cyclomatic complexity of a software program indicates how complex the source code is. This method determined the number of independent paths through a program’s source code. It can guide you, post implementation, as to whether or not a program may contain problems.

Failure Mode Effects Analysis (FMEA)

FMEA is a step by step process used to identify possible failures in design, engineering or deployment of a software product. You start by identifying each function of the software and then identify the modes in which it can fail. Later you look to address them. This process requires a cross functional team for best results.

Delphi Technique

Having run a Brainstorming session, you may be wondering if using the Delphi Technique is overkill. The answer is no. The reasons are many but key to them is that fact that you have relevant skilled experts in the room, the technique is formal and structured and their feedback is anonymous. To start a Delphi Technique you ask your panel of experts to openly discuss the project, defining the problems. Based on this, provide them with survey or questionnaire to get their anonymous feedback, collate & publish their results, solicit feedback and repeat until you have everything you need. A good step by step guide can be found here. Just remember to act on your findings!

Observational gaps in Risk Breakdown Structures

RBS as a Pivot Table

It is advisable to represent your risks in a Risk Breakdown Structure (RBS). (I automate this as a Pivot table in Excel.) This is achieved by categorising risks based on Scope, Schedule, Quality and Cost. I go a step further and create a sub-category related to the major areas / themes within the project. However, I then use this not to observe what is represented in the structure, but rather what is not. If I get the sense that the schedule is the riskiest part of the project, but that is not represented on the RBS, then I know I have gaps. Similarly, if I sense that a certain theme is the riskiest but that is not obvious from the RBS, then I know that I have to spend more time brainstorming on risks related to that Theme.

Residual Risks & Secondary Risks

Having dealt with original risk, a secondary risk can arise as a direct result of your risk response. You need to cautiously watch out for these as they can sometimes be worse the initial risk. The most cited example is putting out an animal trap in order to protect your home (risk), but a family member gets caught in it instead (secondary risk). If you had caught the wild animal in the trap, this could now attract other unwanted animals, this is a residual risk. Brainstorming should tease out these types of risks.

I have also publishing a less detailed Risk Identification article, which you may enjoy more as you weren’t planning on spending this much time trying to uncover loads of risks. Next time, we will review the various analysis techniques that can be used to size and prioritise the risks that you have found.