Are We Measuring Product Success Correctly? - Advice for PMs

Measuring product success is an integral part of any company, without which we cannot operate a product in any way or form. Without measuring success, the whole product vision would simply be based on some deluded platform of presumptions and intuitions which may or may not be correct. However, the question is what constitutes as ‘success’ and what should be measured in calculating how close or far we are to this success?

This question is not an unfamiliar one, and many luminaries in the field have clarified what the success of a product means. Out of these, some of the more well-known personalities include Marty Cagan and John Doerr, who have both tackled the meaning of success head-on in the various notable books they have authored.

To put it simply, success is meeting the product’s objectives, and this is heavily dependant on what the product aims to achieve.

For example, products of non-profit organizations would in many cases differ greatly from a SaaS cloud service. It thus becomes imperative to define the goals of the product to better understand what can be measured to gauge this success.

We need to formulate a general approach within the scope of product management to cover the various objectives of the multiple products out there and those that are yet to come.

Luckily, Doerr already does this for us in his book Measure What Matters, where he introduces the now widely accepted concept of Objective Key Results (OKR). For those of you who may not be familiar with the concept, I would highly recommend reading Doerr’s Measure What Matters. Of course, there may be other manners by which the company’s goals can be defined.

Nevertheless, the definition of success remains the same. Could we reach the goals we set out to achieve within the set time-period? Considering this definition, we must, however, also understand that achieving success cannot only be measured in binary form. Yes, we either achieved the goals set or did not, but by how much did we miss the mark, or when did we become successful.

It is in these assessments where the importance of measuring our success and progress comes into play. Without having the ability to measure our success and progress towards this success we cannot gauge our progress towards success, or identify areas of improvement.

The industry acknowledges this, and nothing I have written so far is relatively new. We already have many different forms of product metrics to put into perspective our KPIs and overall objectives. However, these metrics gauge success in the scope of the product itself, not what it took to achieve the metrics themselves.

This is what this article aims to solve, the disconnect between team metrics and product-specific metrics.

Connecting the team’s performance to the product’s performance itself, and how this bridging the gap between these two metrics can allow better insights to the product management role.

The Value of Traditional Metrics

There is great value in collecting and understanding the conventional metrics of growth and customer satisfaction and engagement. I in no way intend to challenge the value of such metrics in their contribution towards measuring success and influencing product decisions.

There is a multitude of metrics that can be collected, forming a potpourri of acronyms such as MAU and CSAT that we as product managers often come across in our day-to-day activities. Overall, these metrics are aimed at helping us understand the following facets of the product and its users.

• Financial soundness and business growth: Metrics such as Anual Recurring Revenue (ARR) and Customer Acquisition Cost (CAC) help in m measuring how much revenue, losses, and eventual profit the company is making and would be estimated to make. As can be expected, such metrics are usually employed by-products that aim to grow financially or at least evaluate their financial standing.

• User engagement and interest: To measure not only activity but sustained activity. This is aided by metrics such as Stickiness and Retention Rates which can be utilized to check the activity of users using the product. The concept is simple, as user engagement increases and user engagement actions are activated, the more growth you experience and greater revenue is generated.

• User Sentiments and Satisfaction: User engagement metrics alone cannot be the only measure of how successful your application is in solving the customer problem. It may have a lot of traffic, but may not be optimal in terms of UI/UX or other factors to meet the implicit needs of the user. This is where user satisfaction metrics such as Customer Satisfaction Scores and Net Promoter Scores (NPS) come into play which is calculated based on users who are ready to refer your product and those who dislike it.

The metrics referred to above are by no means an exhaustive list. Considering the three general categories stated and the various metrics which fall under them, it is notable that there are numerous ways to evaluate the success of a product. However, these metrics all have one flaw, and that is that none of them take into account the human effort spent in taking the product to the state which reflects the values procured as per the metric calculations employed. It is time we evolve our thought process in calculating business success by factoring in the cost of team health and performance, as this plays a huge role in achieving our milestones and OKRs.

Team Metrics

The feeling of meeting set goals validated from the objective perspectives that the mentioned metrics above provide definitely is an enjoyable experience, but is it true? That is the question we would be asking ourselves constantly in the success of a new product feature or release. What did we lose in achieving this success, or what were the potentials that we failed to tap into?

This is where team metrics should be leveraged to understand how the team operated in building towards achieving the KPI results measured. This is because amazing results can always be achieved, but it does not necessarily translate into sustainable results. After all, the goal of products is not only to be successful within a single period but achieve the prolonged success which comes about by robust growth.

This form of success can only be measured by factoring in team metrics which provide insights into how the team is operating, enabling us to identify risks or opportunities regarding team performance.

For example, a team managed to release a great feature that solved a recurring user problem by building a novel approach within a small period of time. This is truly amazing, and which product team would not want to achieve this.

However, did the team overexert themselves in achieving this level of performance? If they did it can be expected that the team’s performance in the coming sprint or next quarter may not be as it was for this feature. The answer could also be that in fact, the team did not overexert themselves at all, and hence knowing this, the team can collaboratively set the bar of success even higher in their next planning session.

Understanding the value of measuring metrics directed towards team health and performance can hence be leveraged for better product vision construction, roadmap building, and timeline estimations.

Therefore, it becomes imperative that we can measure and incorporate these metrics, especially for product managers in strengthening the commitments that can be made to consumers and other stakeholders across the company.

Understanding this, the next step is to identify these team metrics that can be incorporated into the overall calculations of success. Similar to the product metrics discussed above, team metrics can also be broadly categorized under two scopes which include team health and cohesion and team performance.

Team performance

Team performance can easily be quantified even just using basic measures such as the number of Jira issues completed within a certain sprint or the ability to complete all tasks tracked using burndown charts. Apart from the basic metrics that are available and quite intuitive, there are more sophisticated metrics that have come-up over the past few years considering the shift towards new trends in software development. One such shift is DevOps, and hence metrics associated with the shift can add great insights. As per DORA’s highly reputable 2019 State of DevOps report, five metrics can be used to calculate the team’s performance in both development and operations. These include the following:

• Lead Time: the time taken to go from code committed to code in production.
• Deployment Frequency (DF): How often does the team perform deployments.Mean Time to Resolve
• (MTTR): The time it takes to recover from a disruption.
• Change Failure Rate (CFR): The ratio of changes to the disruptions experienced.
• Availability: The amount of time team owned services are accessible.

Team Health

These metrics aim to calculate how well the team is working to achieve their goals and especially how each individual is poised as per their role in the team. Unlike team performance, team health cannot always be easily quantifiable as it tries to measure the more emotional and inner state of a team member.

This is quite an abstract concept and is only recently being surfaced as its importance is being realized. This is because certain periods of the product’s growth may be extremely demanding, or not challenging enough to keep the team members and the team as a whole engaged or motivated. Nevertheless, there are well-developed exercises that teams can perform in measuring the team’s health. This is something that companies like Atlassian have been focussing on, promoting team improvement and health metrics such as the Health Monitor Playbook.

Measuring both team performance and team health metrics in the scope of success would provide the much needed holistic approach in gauging our progress. By understanding that the team is experiencing burnout or issues, similar to considering the churn rate in goal setting, we can attain more realistic approaches. These metrics would also provide an additional dimension when measuring risk as a team with degrading performance and health can result in the project being derailed. Additionally, by having a full understanding of the situation backed by all relevant metrics, the right contingency plans can be put in motion to ensure that success is achieved as expected, and most importantly, sustained.

Conclusion

The value of product-related metrics such as those measuring financial growth, customer satisfaction, and engagement has proven vital for product managers throughout the years. However, as our understanding of how software products are built improves, we can see that sustained success is not accurately measured unless team metrics are juxtaposed accordingly.

By considering these metrics, we can finally close the last gaps between product managers and the engineering teams, ensuring that we as a whole are planning for sustainable success.