Assessing Project Health
Introduction: Different Metrics for Different Purposes.
When we are discussing metrics within project management, it is important to look at the big picture. In this respect, all aspects of portfolio, program, and project management, as well as the organization performance, should be considered. From this starting- point there are the following broad categories of metrics:
- Change management metrics:
- Process Maturity (based, for example, on the Organizational Project Management Maturity Model, [OPM3®], of the Project Management Institutefrom PMI)
- Learning
- Program and project process compliance metrics
- Effectiveness: organizational project success
- Operational metrics:
- Portfolio health metrics: effectiveness
- Program health metrics
- Project health metrics
- PMO scorecard
Each of these categories look at project management in a different way, and it is important to establish metrics for each. Change management metrics aim at assessing how far the organization has gone in implementing new processes and reaping their benefits. Operational metrics are used on individual projects/programs/portfolios to assess how well they are performing. The PMO scorecard is a combination of both previous metrics that focuses on the objectives of the PMO itself.
In this article, we will focus on project health metrics that enable management to make critical decisions thatto help the project manager and his or/ her team succeed. Without objective metrics, it is indeed sometimes difficult to judge whether a project is actuallyreally on track.
If you want to know more about the other metrics mentioned above, please refer to the article “PMO metrics,” part of the PMO Accord built by the PMO SIG from PMI, authored by Nicolas De Dobbeleer and Kjell Rodenstedt.
Symptoms Related to Ill-Controlled Projects:
The following are some of theHere follow a number of typical symptoms related to the way projects are monitored:
- Cost/ benefits analysis: the condom or “Kleenex” syndrome
Often the cost and benefits of the project are calculated once at the beginning of the project but never revisited afterwards (used once, then thrown away). This way projects are never canceled when their return on investment or payback changes.
To the question “Can we spend €500k Euros more on the project?”, the answer can easily be, “Sure, now that we have started we can’t stop it,”, whereas in fact a good answer would be “It depends. How much in additional savings per year will it bring back?”.
- Planning/schedule: a moving target
Similarly, the initial schedule is often built only once (in Excel) to get project approval, then left aside. And even when schedules are maintained, updates are often made only every 3 to 6 months, which makes them useless, as they do not reflect reality in the meantime, and therefore cannot be used to make decisions on the project.
The question is whetherif this common behavior is a cause or a consequence of the fact that schedules are not perceived as real management tools. A common comment is “Why spending time updating a planning since it will change anyway?”. The point is that a poor planning creates the need for repeated updates and thereforehence a waste of effort, whereas a good planning allows prioritizing efforts. The baseline serves as an approved reference against which progress can be measured and deviations analyzed and minimized, and which normally remains stable over the life of the project.
- No progress quantification: to be… or not to be… optimistic ?
A typical status report shows the following: tasks completed, - issues faced, - next steps. While useful, these elements are not sufficient to make the right decisions.
A typical storyline without quantification of progress look like this: on the third3rd month after the project has begun: “no problem”;- on the sixth6th month: “no problem”; – on the ninth9th month: “the project is several months behind schedule, but will get back on track.”. Similarly, dashboards with colors only (green-yellow-red light), without solid metrics behind them, are too subjective to trigger appropriate actions.
For example, few projects use timesheets (only projects that invoice directly to the business), and even then the effort cannot be compared with the real progress made on tasks.
- Wrong progress quantification: eyes wide shut
Is the progress measured as the percentage of budgeted effort spent?
Example: “We have spent 200 out of 1,000 man-days, so we are 20% complete.”.
The real question should be “What have you really delivered so far?” or “How late are we? What are the necessary corrective actions?” “
Too few projects quantify progress directly on deliverables.
- Status reporting: priority #100 on the project manager’s list
Some project managers consider reporting as an administrative task, but one could argue “Is taking the time to review the progress and identify issues not part of the project manager’s job?”. As a result, they send their reports late, which makes the reporting less accurate, and a vicious circle starts, where information is less and less reliable.
- No decision making: do it yourself !
Sometimes the same issues, often related to resources, are reported on several consecutive months. What is the role of management in this matter? Sit and listen? Or act to help remove the obstacles?
Real decision making encourages project managers to report timely and accurately to management.
- Scope creep and lack of quality standards: to do… “later” and “good enough”?
One can hear Once that the project is late, we might starting hearing “Do we really need this deliverable?”, or
“We will do it after the project, as part of the continuous improvement phase.”.
When no quality standards are defined, it becomes subjective to assess what is done or not, and it is even worse that is whenif formal review of deliverables by management are even not conducted.
Diagnostic
A loose control of the project leads to more likely failure. Poor control is mainly due to the following:
- Lack of timeliness and quality of reporting
- Inconsistent and incomplete metrics
- Lack of decisions/corrective actions based on the metrics
We describe in the next paragraph how these three main issues can be solved.
Remedy/Solutions
Status Reporting
All projects identified should be reported upon at least on a monthly basis (it could be more frequently for projects in a critical phase of development). Projects status reports using a single template are expected to be posted by the project managers by the xth day of each month. They in turn need to get the information fromform their team members a few days before that deadline.
A project management office (PMO) can play an important role of quality assurance with respect to the status reports and summarize the information into a dashboard. The dashboard is then not only a set of subjective flags (red/yellow/green) but a well-balanced and objective representation of the overall health of the portfolio of projects, allowing a fair comparison between projects.
The dashboard can be automatically created based on the pre-defined status report format, reviewed by the PMO for consistency and published by the x+yth day of the month. The PMO should ask the project managers to review the dashboard before broader communication to avoid misunderstandings and increase buy-in.
Project managers should be encouraged to deliver the status report on time by adding theseis criteria in their performance review. Indeed, the timeliness of the report is very important for the appropriate decision-making process (e.g., timely resolution of escalated issues). Project managers should also be trained on the metrics so that they share a common view on the criticality of issues with management.
Eight Metrics
We believe that eight main categories of metrics cover the most important aspects of a project. Within each of these categories, a number of sub-metrics can be defined. These sub-metrics can then be combined into one to create the dashboard.
- Scope.: Scope changes should be reviewed and approved by the steering committee of the project within a certain timeframe. For example, the metric can be for example:
- The number of scope changes with an impact greater than 10% of the project budget still not approved. Alternatively, a threshold for the timing of approval depending on the size/criticality of the scope change can be combined to translate this into a global health color-code.
- Time.:
- The most widely used metric is SPI, Schedule Performance Indicator (linked to the concept of Earned Value), which compares the progress made with the baseline. Ideally, the SPI should be higher than or equal to 1 (which means the project is on time or ahead of schedule).
A recent research by Professor. M. Vanhoucke shows that from a percentage of completion of 50 % already, the traditional SPI, based on the earned value measured as a cost (on the typical S-shaped curve showing costs over time, the schedule variance is shown as the difference at instant t between earned value and planned value, on the cost axis) is much less accurate than a SPI based on the resulting time difference calculated from the earned value itself (on the typical S-shaped curve showing costs over time, the more accurate schedule variance is measured as the difference between earned value at instant t and the equivalent planned value at instant t’, on the time axis). Please refer to the book “Measuring Time” (still to be published; check the URL http://www.protrack.be/protrack_measuringtime.php) by Prof. M.Vanhoucke for more details. The results of the research are too comprehensive to be covered here.
It is in reality difficult to have a perfect plan; therefore, usual measurements can be between 0.7 and 1 for less mature organizations. The threshold for considering the project in danger has to be adapted in function with regard toof a number of factors, including the industry, the type of project, the maturity of the company, etc.
- The percentage of critical tasks (i.e., on the critical path, the longest path that determines the duration of the project) that are overdue and actual project duration versus baseline?
- Cost:
- The most widely used metric is CPI, Cost Performance Indicator (Earned Value) which compares the actual cost with the baseline cost of work performed. Same comment as for SPI regarding the interpretation of the metric.
- How much has been spent so far on the project (actual cost of work performed- AC) versus budget (planned cost, PV) per project period and phase? This can help improve estimates on the next projects.
- Benefits over the product lifecycle.: It is important to track the benefits of the project over its full lifecycle. It could happen that after a feasibility study the project is shown to not bring the expected benefits and should be cancelled at the next decision gate. These metrics should be discussed with the finance department since they define the standards in that field. A number of metrics are possible, such as:
- ROI (Return On Investment— , e.g., benefits divided by investment over the first 5 years after the project starts, for example).
- Payback (the time that it takes to recover the project investment).
- Product quality.: Project deliverables needhave to be reviewed and formally accepted.
- How long does it take to get the approval of project deliverables?
- The percentage of quality standards not reached without rework (based on a quality function development study).
- Resources:
- The number of times that a department generates delays in the project portfolio.
- The percentage of team members who repeatedly (more than x times) deliver late and the reason behind it (issue related to deliverable)
- The number of times that the unavailability of resource delays critical tasks.
- Stakeholder satisfaction:
Three main categories of stakeholders are critical:
∙ Is the customer satisfied? After all, the customer is the one receiving (and usually paying for) the deliverables of the project. He should be the most important person to listen to.
- Are there any personal issues within the team? Who is not satisfied with being a member of the team? Why? What can we do to solve this situation? It is of the upmost importance to keep the team motivated.
∙ Is management / the sponsor happy with the results so far? The sponsor is the one paying for the salary of the team and giving authority to the project manager.
These are more subjective than the previous metrics but are nevertheless important to balance the perception of the whole project. The answer can be binary (yes/no) or a scale from 1 (completely unhappy) to 5 (fully happy), for example.
- Project management quality.:
Project management quality focuses on the process of managing the project, rather than the quality of the product itself. A number of criteria can be used:
- Is the project managed according to the plan?
- Is the project schedule regularly updated and the status report timely and accurate?
- Are changes managed and documented?
- Is there a regular follow-up of the action items identified during the reviews?
- Is there a mitigation plan and/or contingency plan developed at least for the most important risks? (The metric can be the availability of mitigation and/or contingency plan for risk rating >2.5 (= risk probability * risk impact/ 25) using a scale of 1 to- 5 for impact and probability).
- Does the project team perform the QA activities as defined in the quality and project plans? These activities can be, for example, formal inspection and handover of internal project results from one process to another, or quality audits.
If any more than one1 answer is “no,” this should bethen an alarm toshould trigger corrective action.
Decision- Making Thanks to a Good Dashboard
The dashboard consolidates all metrics and shows traffic lights and an additional trend arrow to show the evolution from last time. A one-page wide format is recommended so that readers can have a good overview of the portfolio.
It is a good practice to define automated rules for the colors of the traffic lights so that the PMO does not need to start from scratch, but of course the colors should manually be reviewed after automatic calculation, since it is extremely difficult to define a rule that applies in all circumstances.
The thresholds for colors will depend on the degree of maturity in project management of the company. For example, for earned value, the SPI and CPI indicators should ideally be 1 or above. But in reality, it depends on how optimistic the project manager is or if estimates from previous projects are available. It is not recommended thatto define rules be defined that are too strictly, because thisit will discourage project managers to participate and it will decrease the contrast of the dashboard by showing too many criteria as red or yellow.
The usual colors are green for OKok, yellow for requiring management follow-up, and red for requiring immediate action from management. Blue is sometimes used for approved projects for which reporting is not active yet. White color can be used when the status report is not available.
A global project health color can be derived from the other metrics, based on a weighted sum of the color values (e.g., 0 for green, 1 for yellow, and 3 for red). Each metric is given a weight (e.,g., 6 for time/cost/quality, 5 for resource, 4 for scope/benefits/ project management quality, and 5 for customer/team/management satisfaction).
Then the project health could for example turn red, for example, if the total score is higher than 40, yellow between 15 and 39, and green below 15.
The dashboard should also contain basic information such aslike the name of the project manager, the name of the sponsor, the current project phase, the next deliverable to achieve, the budget, the internal man-days spent, the progress made so far. , …
It is also useful to have a field where the PMO can provide recommendations to project managers to improve the quality of the project management on individual projects.
Conclusion
In order to properly assess project health and make appropriate decisions to ensure that projects stay on the right track, three main elements are necessary:
- Regular status report including update of project schedule
- Appropriate metrics combining hard (quantification of progress, etc.…) and soft metrics (people satisfaction)
- Dashboard enabling appropriate decision making
Examples of detailed metrics can be found in this article. These as well as the format for the status report and dashboard have to be adapted to the culture of the organization. Standard metrics allow the organization to compare projects with one another.
