Key Performance Indicators (KPIs) are a well established and important feature of many high performing teams. They are used to monitor significant metrics which can then provide leadership with valuable insights into the health of an organization or team.
At each of my three most recent jobs, I’ve worked through a book called Traction by Gino Wickman. In it, there’s a chapter all about Data which emphasizes the importance of narrowing our focus to what really matters. It’s impractical and ineffective to try and monitor everything that “matters”. While it isn’t always easy to do, settling on a smaller set of figures to elevate and report on constantly does consistently produce real progress. I strongly recommend this book and this chapter in particular.
Anything that can changes over time and can be measured can be a KPI and every company has to decide for themselves which PIs are K (does that work???). One company may decide that “average time to first reply” is a support KPI whereas another may consider “average number of replies” or “total negative ratings” to be paramount.
KPIs can be established for any role and any department. Even roles not commonly assigned KPIs can have them. There’s always something measurable.
However, when I see or hear people talking about KPIs, it is very often sales and marketing related metrics which become the focus. It is very straightforward to look at obviously important and easily accessible data like revenue, sales, profit, site traffic, and so on. Most of us are also looking at slightly more nuanced numbers like churn, free to paid conversions, customer lifetime value, average order value, site conversion rate, and so on.
In the future I’d love to pen some posts about KPIs for every common business function as its something I’ve spent a lot of time thinking about. For now, I want to focus on one in particular: Software development.
Software development KPIs are very interesting to me because the function is far removed from sales and almost all of us have dev teams. For a function this ubiquitous, it surprises me how infrequently I encounter WordPress dev teams seriously utilizing KPIs. I’ve even had some people challenge the idea that developers should have KPIs which I am a bit baffled by. The idea that any team should be exempt from performance indicators, especially when it is the team with the most expensive staff (watch out, don’t get me started!), makes zero sense to me.
Engineering KPIs are definitely a thing but so many of them are not helpful in a WordPress context. If you research the topic you’ll discover many traditional KPIs for engineering teams such as “cycle time” and “throughput” which either don’t make much sense for WP projects or are just not useful to monitor. I’m very interested in learning more about which metrics pertaining to software engineering are being analyzed by WordPress companies. I’ll ask again for comments at the end of this post but for now, I’m going to lay down a list of a few metrics which I believe can be useful for WordPress dev teams to track:
Feature to Bug ratio
Possibly better referred to as Enhancement vs Fix since not all improvements are “features”. The idea here is that if we can break all issues into two groups, improvements and fixes, we can keep an eye on the number of issues we’re closing from both. A team could agree on a target ratio and try to consistently come as close to it as possible.
For example, a team could make this very simple and target a one-to-one ratio, meaning one improvement for every correction, or one feature for every bugfix. This ensures the software is constantly getting polished and also moving forward.
I’ve been in situations where this ratio was way off and have seen what problems it can cause. If features are too heavily emphasized, we struggle to keep users because we aren’t fixing problems. If all we do is fix bugs, we lose ground to competition.
I’d add that this number is probably largely dependent on the stage of a business with a bias towards features in the beginning and towards polish later on. A product could start with a 5:1 target in year one but have a 1:2 target ratio in year five, for example.
Regressions
Regression bugs are created when something that formerly worked is unintentionally broken by a newer change. They tend to indicate a failure in both automated and manual testing processes. If regressions are high, you might want to invest more in unit tests and QA.
Open Source Contributions
WordPress is open source software and continues to live on and improve because of contributions from volunteers in the community. We’ve all benefited from the work done by thousands of open source contributors in the past and so it isn’t unreasonable for us to consider giving back. Many dev teams in WordPress allocate time and resources towards improving open source projects (not just WordPress core). This is a pursuit which can be measured and have goals defined.
Issue Open vs Closed rate
This is a very simple but useful number to monitor. It’s a generic measurement of how efficiently a team is at keeping up with a project’s needs. In many cases, the target might be 1:1 which means that over a given time period, for every new issue that was opened, the dev team resolved one other issue. When the open number is greater, the size of the backlog is increasing and when the closed number is greater, the backlog is decreasing.
Average Days to Issue Resolution
If we look at how many days take place between the time an issue is opened to when it is resolved, and do this for every single issue, we can get an average which is a very interesting number to look at when planning for the future. Other teams such as Product, Marketing, and Support, all appreciate knowing this number and it can factor into their plans.
Reversion Count
Reversions are the recalls of software development. It’s the code which gets shipped but then shortly after has to be removed, either permanently or just until it gets improved. Reversions signal problems but the sources can be many. They can come from failures in testing, failures in design, and failures in feature validation.
Test Coverage
Expressed as a percentage, test coverage reveals how much of a codebase is accounted for in its suite of unit tests. These automated tests are instrumental in preventing regressions and thorough test coverage is one of the most effective strategies for minimizing bugs getting shipped.
100% test coverage is often hard, but not impossible. Dev teams which emphasize testing from day one have a much easier time maintaining a high test coverage percentage compared to those who try to develop tests long after the fact.
Ratio of Issues Resolved on Free Version vs Paid Version
For anyone operating on a freemium model, it can be useful to monitor the amount of effort being put into the free version of a product compared to the effort going into whatever makes the money. Again, the correct ratio is going to be different for every product but it will always be possible to skew too far towards one or the other. Dev teams can ensure both versions of a product get the focus they deserve by monitoring the number of issues resolved (and probably other basic productivity metrics) over time.
Issues Opened by Source
Not exactly a pure KPI but this can be a very useful pie chart to look closely at on a regular basis. “Sources” in this context are the reporters of an issue. For many of us, the lifecycle of code begins with the dev team, then goes to Quality Assurance, then often to beta testers, and then finally to actual users. Exact sources will vary but there are always multiple steps in the process.
Looking at this is important because we need to understand when issues are being discovered and reported. How many issues are being caught by the developers? How many issues are caught by whomever is doing QA? How many are reported by beta testers? Finally, how many are reported by users?
We want that last number to be as small as possible. If too many issues are being discovered late in the process than work needs to be done to improve the testing earlier.
Version Breakdown
I don’t really believe this is a great KPI per se, but it is something that WordPress dev teams should be very mindful of and taking some degree of ownership over. It’s true that if we’re shipping plugins and themes, we can’t control whether people update or not. But we can take responsibility for removing any barriers from upgrading.
WordPress admins who run an update that breaks something will very often revert to a previous version (or deactivate the plugin entirely). It’s our job to be as thorough as possible with our testing, both automated and manual, to minimize occasions like this. This is especially important because it is all too easy to lose user trust. Ship a glitchy release one time and for years some of your users will fear the update button.
There are also instances in which version changes necessitate upgrade routines which require action on the part of the site admin, like manually running a database upgrade or updating some new settings. These are highly undesirable, though sometimes unavoidable. It sucks to have to do this because some users will get nervous and hesitate, some will procrastinate because they think they don’t have time, and others still will find a way to screw up the process. It can require some creativity and extra effort to avoid these situations but hey, that’s our job. And oftentimes it’s well worth it to avoid creating a barrier to upgrade.
Contributors
For those of us maintaining a project which is all or partially open sourced and accepting towards community contributions, it can be useful to track the number of unique individuals participating in development. If this number is too low, some investment into evangelization may be warranted in order to spark some community engagement.
Setting a goal of some kind for an average number of contributors and making an effort to respectfully solicit and nurture a development community can be hugely beneficial for open source projects.
There you have it. A handful of measurable figures directly affected by software developers which could be used as Key Performance Indicators.
Does your dev team use KPIs? If so, which ones? Thanks for sharing!