Metrics and KPIs in the Agile World | David Tzemach
Metrics, when used correctly, are the most efficient and effective way for Agile development teams and their management to promote continuous improvement, understand the project’s progress over time, and gain information on aspects of the project such as team productivity, value to the customer, and process improvement.
The Uses and Benefits of Metrics
Many stakeholders are involved in an Agile project: the development team itself, customers, senior management, and project management. Each would like to see how different aspects of the project are going.
Metrics enable the business and especially the development teams to view the time spent on different quality activities and monitor quality improvements over time. They provide transparency on the project status and the remaining effort for completion. Metrics can assist in determining how effective and efficient the development process is and assist the business in understanding its real ROI. They can help identify the project risks to create a mitigation plan and determine which practices are working and which are not.
Metrics and Key Performance Indicators (KPIs)
Can you explain the difference between KPIs and metrics? Hopefully, the following paragraphs will make it clear.
What is a metric?
A metric is a quantifiable measure used to monitor and assess the status of a specific business process or software characteristics that are either quantifiable or countable. The use of metrics is essential because they provide a comprehensive vision of all trackable areas, while KPIs provide more in-depth details in the area we are measuring. For example, a metric can monitor site traffic, whereas a KPI will focus on specific areas such as access time, uploads, or content downloads.
What is a KPI?
A KPI is a measurable value used to determine how effectively an organization achieves vital business objectives within a specified period against acceptable norms. The organization uses KPIs to evaluate how well they have achieved their targets.
KPIs provide numbers that are beyond feelings. For example, if a team has an incredible feeling of success, the way to determine if they are correct is by using measurements and comparing them over time.
The Agile environment strives to establish transparency throughout the entire development effort—KPIs help evaluate progress and set the baseline for documenting business performance.
The letters, incidentally, stand for Key Performance Indicator.
The Need for Metrics In Agile Projects
We can all agree that using metrics in Agile projects is a necessity. If you are still not convinced, here are a few more reasons:
Continuous feedback loops - Metrics allow Agile teams to get honest feedback about their work without making assumptions. A good metric helps the team understand when they are getting off track and what should be changed to return to it.
Metrics increase focus - The danger in using metrics is that they can impact people’s behavior when they know their work is being measured. However, if used correctly, metrics ensure teams focus on doing the right things rather than wasting their time on less impactful activities.
Motivation - Metrics can be a tremendous motivating force by showing the team their progress and improvement.
Input for retrospectives - The metrics results can be used as critical input for team retrospectives to examine what worked well and needs improvement.
Keep track of quality - Metrics allow teams to understand the quality of their deliverables and what they need to do to improve them. Moreover, metrics can help the team receive alerts regarding quality degradation they cannot see in day-to-day activities, such as:
Degradation in code coverage over time.
An increasing amount of technical debt.
Less test coverage compared to the original estimation.
Bugs increase for a specific period in the project.
Questions to Ask Before Setting a Metric
There is a fine line between metrics that contribute to the project and those that become a burden and are not worth the time spent to collect their data. To determine the real ROI of a metric, use the following questions:
Is it clear enough? Can it be easily communicated to the organization?
Do we have the tools or people who can provide the information?
What is needed to maintain the metric over time?
What is the problem it will help me resolve?
Does it add real value to the team?
Is it worth the investment?
How To Create Metrics In Five Easy Steps
After years in the software industry and in Agile projects, I have defined a relatively simple process for using metrics in Agile projects.
Step 1: Set the baseline for the project
Instead of concentrating on the metrics you currently have, or the different aspects that you think are not working for you, start the project with a clear mind, focusing on the following items:
What is the goal you want to achieve by using metrics?
What are the tools you will use for gathering and monitoring your metrics?
What are the main areas or aspects of the project you want to improve?
Step 2: Determine the metrics that add the most value
Determine the metrics that the team will use throughout the project. What metrics will give you information that is genuinely essential to succeed in your project? These metrics should follow the rules described earlier in this article to ensure they add real value.
Step 3: Publicize the metrics and generate feedback
Communicate the metrics to the teams that are being measured. To harness the team to the process, it is essential to clarify what results reflect success and allow the team to share their feedback and concerns.
Step 4: Start measuring.
Now that the team has shared their feedback and is clear on what metrics the organization will use, it’s time to gather information about each metric.
Step 5: Refine over time.
There is always the chance that metrics considered crucial at first are no longer relevant or that metrics that were thought to be easy to measure suddenly demand an effort that makes them impractical.
When this happens, the organization must refine its metrics to add real value and contribute to the continuous improvement process. Remember, metrics are valuable so long as not hurt your teams.
The real value of the metrics will come with time. Once you have anywhere between four and six sprints of data generated by a specific metric, you will be able to conduct a more precise analysis of your findings. As the data grows, you will see trends in your data and learn how key parameters evolve through time.
Scope of Metrics in the Scrum Environment
Now that the challenges and efforts involved in creating metrics are clear, we should take a closer look at metrics in Scrum projects.
Organizational metrics
The metrics at this level are created by senior management and reflect the organizational goals, vision, and overall direction. The organizational vision has a decisive impact on the effectiveness of the metrics, and an organization without a clear vision will have problems creating metrics.
Due to the importance of these metrics, it is highly recommended that they be communicated to all stakeholders involved in the project.
Project metrics
Project managers and their departments create metrics used to measure different aspects of the project (delivery time, project costs, etc.).
Sprint metrics
These metrics are usually created by the Product Owner, project manager, and sometimes the team itself. They are used to determine team performance and effectiveness throughout the sprint. The metrics used at this level can and should be changed every few sprints based on what is happening in any given period.
Team metrics
These metrics are an extension of the sprint metrics and are defined by the team based on what they think is essential for their self-improvement. Scrum teams often neglect these metrics as they do not want to add more metrics to those defined at the previous levels. This is a mistake. Team metrics are essential because they help the team improve and increase the overall transparency of their work. Here are a few metrics teams may find helpful at this level:
The amount of time that is going towards External support.
Team Happiness score.
The number of external interruptions to the team’s work.
The number of effective meetings.
Top Metrics for Agile Projects
Let’s review some commonly used metrics for Agile projects.
Agile health metrics for Agile teams
The metrics in this list are used to monitor your team's health. These metrics have become popular in recent years, although some of the metrics in this list are not directly related to the actual development process.
Team happiness – This metric allows the business to track and identify any HR issues affecting the team’s ability to work at the highest level. That said, there is no natural way to measure happiness. However, with some creativity, we can find productive solutions.
Impediment removal per sprint – This metric is based on the impediments board that is part of the retrospective meeting. Its primary purpose is to review the average number of impediments handled per sprint. This metric reflects a team's ability to mitigate impediments that influence their day-to-day activities.
Meeting the sprint goal – As explained earlier, the sprint goal helps focus the team and ensures coordination between stakeholders. Therefore, it's logical that we will want to see how often the team has accomplished its sprint goal. By using this metric, we will gain a qualitative assessment of how good the team is and whether the team can deliver what they committed to or not.
Agile quality and testing metrics
The Agile quality metrics are used to determine the quality of the products built by the team and whether the team meets the quality standards.
Bugs deferred – Agile involves rapid releases that put the team under a lot of pressure to deliver. This pressure often leads to impaired judgment and deferring identified bugs to future sprints as a form of technical debt. This is one of the main reasons we want to measure the number of deferred Bugs that the team generates per sprint to ensure we don’t compromise on the quality of our deliverables.
Escaped Bugs – Escaped Bugs are a powerful but straightforward metric: the number of bugs for a given release is found after the release date. This indicates what the team needs to improve to avoid similar Bugs in the following releases, which is priceless. In addition, the team will have information on areas that were released with poor quality or that should be tested more thoroughly. They will be able to add tests based on the escaped Bugs and gain insight into the effectiveness of the risk assessments made, primarily when the team used risk-based testing as their testing method.
Escaped Bugs fix time – If a bug is found in the customer environment, providing a solution as early as possible is essential. This metric establishes the baseline for the average time it takes to find a solution.
The number of failed deployments – Due to the many releases involved in Agile development, it is crucial to understand how many releases were deployed successfully in customer environments and whether they fulfilled the Agile value of building shippable software.
The number of escalations allows the business to understand the quality of the deliverables deployed in a customer’s environment. The number of escalations should be measured by analyzing the total number of escalations and comparing that to the conversion rate of escalation into product bugs (which indicates a quality issue).
Technical debt – Technical debt affects a team's ability to deliver software. If the team generates technical debt, they will also need to clear it out. During this time, the team focuses on stories that do not add direct value to the customer, contradicting one of the essential values described in the Agile Manifesto. Two metrics I like to use with my teams are the average amount of technical debt created per sprint and the average amount of technical debt cleared per sprint. These metrics allow teams to understand what they should improve to reduce this debt.
Violations of coding standards – Coding standards may differ from one organization to another. Nevertheless, this metric measures the coding standard violations that led to poor quality deliverables, such as check-ins without code reviews, the creation of poorly documented code, and so on.
Agile productivity metrics
Agile productivity metrics assess the productivity of Agile teams in various aspects of % completion of features, stories, and tasks.
Sprint burndown – The sprint burndown chart shares how many stories points the team completed during the sprint and how many are still left.
Release burndown – This metric is focused on the “big picture,” allowing Agile teams to track the overall progress of work and see more than what is displayed in the sprint burndown. By using this metric, the team can be made aware of any productivity degradation.
Lead time – This metric tracks the time it takes for a team to deliver a specific business requirement from the moment of inception until development completion. This metric surfaces the delay factors and the actual productivity of the development process.
Cycle time (control chart) – Cycle time is a subset of lead time; it is used to measure the time from the moment a task is added to the team’s backlog (with “in progress” status) until it's been completed by the team (“done” status). The result you want to see is that your cycle time does not exceed half the sprint length. If it takes more than half a sprint to complete a task, then the task is either too big or too complex.
Team velocity – Velocity measures the number of stories delivered on average per sprint (for at least five sprints and only based on delivered stories). Using this metric, we can predict the team’s deliverables in future sprints.
The number of external interruptions – External interruptions hinder teams from focusing on their work, causing them to fail to deliver on time. This metric focuses on the number of interruptions and their source, enabling us to find ways to reduce them in future sprints.
Work in process/progress (WIP) is one of the most important metrics used in the Kanban framework. It tracks work items that the team has started but has not yet completed and their status. This metric allows the team to improve their workflow and ensure they do not commit to more work than they can deliver, based on the number of work items in progress.
Frequency of changes – This easy-to-use metric measures the number of changes made in the sprint scope once it has started. This metric highlights how often the team changed their focus within the sprint, which of course, reduces their ability to deliver on time.
Cumulative flow – Cumulative flow is a Kanban metric that provides a high-level view of the status of tasks in a release or sprint, possibly across multiple teams. The main power of this metric is its relative simplicity to implement and track, and it provides a great visual of all tasks in any of the workflow's stages. Using cumulative flow diagrams quickly identifies bottlenecks and catches problems in the ongoing development effort before they result in delayed delivery.
Agile testing metrics
Agile testing metrics assess activities such as test coverage, test efficiency, and testing gaps. These metrics can be used to improve testing efficiency.
Automated vs Manual test coverage – This metric measures the percentage of tests covered by automated tests. By narrowing the manual effort, the team can increase their test coverage, reduce the time needed for regression and increase the overall quality. As you will see in the last volume of this book, automated frameworks are crucial for testing in the Agile environment. This is one of the main reasons for a team to measure the automation rate of their tests.
Code coverage – The percentage of code covered by unit tests can be measured by the number of branches, statements, conditions, or functions executed as unit tests. This number can be determined automatically as part of the creation of every build and provides raw insight on how much of the codebase has been tested. This metric cannot be used as a single indicator of high or low quality; however, it is still a good indication of the overall direction of your testing efforts.
Bugs per test category – This metric measures the number of bugs found. This can help understand the product's vulnerability in critical areas like performance, scale, UX, UI, usability, and security.
Bugs per testing layer – Any project contains different testing layers, such as unit, component, and end-to-end. You may want to measure which testing layer provides the best ROI for the overall testing effort. This simple measurement can uncover some of the essential information needed to increase the effectiveness of the testing effort. In addition to seeing which testing layer provides the best ROI, you can determine which testing layer should be modified to increase ROI and whether the test strategy needs to be changed.
Test execution time (automated/manual) – This metric clarifies the testing effort the team needs before releasing the product. The measurement should focus on all testing layers (unit, component, etc.) to allow the team to understand which layer consumes the most considerable execution time and which layers need to be extended or reduced.