Selecting Software Development Metrics

• is a consultant specializing in improving software development and delivery methods. We spoke to him about software development metrics — the factors to consider when selecting metrics, examples of useful metrics as well as common mistakes made in applying them.*

Read more interviews with some of the world’s best developers on * or follow us on or follow us on .*

“I don’t actually find metrics to be an interesting topic,” says Nicolette. “But I think it is a necessary thing. It’s part of the necessary overhead for delivering. If we can detect emerging delivery risks early, then we can deal with them. If we don’t discover them until late, then we’re just blindsided and projects can fail”.

Through his work Nicolette has worked with countless technical managers, but he’s observed that “a lot of managers, team leads and people like that, don’t really know what to measure”. What usually happens is that “when people adopt a new process or method”, usually one of two things happens, says Nicolette. Either, they’re “only using the process in name only, or they’re trying to use it but they’re not used to it yet, and the metrics don’t quite work.” Or despite changing methods, they continue to measure in the same way — “people tend to use the measurements that they’re accustomed to. They’ve always measured in a certain way, now they’re adopting a new process. They keep measuring the same things as before.” So the metrics don’t quite match the process and they wrongly interpret a problem with the process itself.

Another common mistake Nicolette has noticed with development metrics, is that “people either overlook or underestimate the effects of measurement on behavior. We can be measuring something for some objective reason, but it causes people to behave differently because they’re afraid” that it will negatively impact their “performance review.” So Nicolette says, “I think we have to be very conscious of that. We don’t want to drive undesired behaviors because we’re measuring things a certain way. That not only breaks morale, but it also makes the measurements kind of useless too when they’re not real.”

Nicolette suggests that when it comes to metrics, all we really need to do is “look at the way work actually flows in your organization and measure that.” Now, of course, that’s easier said than done. But he asserts that there are really just “three factors to judge which metrics are appropriate.”

“The first factor is the approach to delivery,” says Nicolette. There’s traditional, which is where you “try to identify all the risks in advance… lay out a master plan, and you follow that plan.” Or there’s another type, adaptive, where “you start with a direction and an idea of how to proceed. Then you solicit feedback frequently so you can make course corrections.” According to Nicolette, the delivery type, “traditional versus adaptive… has the biggest impact on which metrics will work.”

Nicolette says that “the second factor to look at is the process model.” “Nobody does anything in a pure way, but if you boil it down I think there are basically four reference models we can consider. One is linear… the canonical steps that we go through from requirements through to support. He says that “the next one would be iterative, in which you revisit the requirements multiple times and do something with them. The third one that I identify is time-boxed. It’s really popular nowadays with processes like Scrum and so on. The fourth one is a continuous flow. This is popular now with the Kanban method, and it’s also being adapted into Scrum teams quite a lot. And it’s where we’re really interested in keeping the work moving smoothly.”

But Nicolette is a pragmatist and knows that any “real process is going to be a hybrid of these, but it’s been my observation that any real process will lean more toward one of those models than the others, and that’ll give us some hints about what kind of metrics will fit that situation,” he says.

Lastly, the third factor Nicolette says impacts which metrics are relevant “is whether you’re doing discrete projects or continuous delivery”. “I look at those three factors, and based on that you can come up with a pretty good starter set of things to measure, and then you can adapt as you go from there.”

To understand how these factors can be applied, here are some example scenarios, detailing the relevant metrics.

Scenario: An agile team, working in short sprints, who are struggling to ship a new product.

Nicolette says, that “if they’re using Scrum basically correctly, they could probably depend on the canonical metrics that go with that, like velocity and your burn chart. You might look for hangover, that’s incomplete work at the end of the sprint. You might look for a lot of variation in story size, when you finish a story in one day and the next story takes eight days.” Interestingly, “you can always use lean-based metrics because they’re not really dependent on the process model… so they could look at the mean cycle times as well as the variation in cycle times and get some hints about where to go from root-cause analysis. Metrics don’t tell you what’s wrong, but they can raise a flag,” says Nicolette.

Scenario: A hybrid waterfall-agile team working on a long-term project, wanting to know what’s possible to deliver by a certain date.

“To know what’s possible to deliver you can use a burn chart, burn up or burn down as you prefer,” suggests Nicolette. This would help them see “according to their velocity, how much scope they can deliver by a given date. Conversely, you could see by what date approximately they could deliver a given amount of scope. It depends on what’s flexible. In this kind of a project, usually neither is flexible, but at least you can get an early warning of delivery risk.”

“One thing that might surprise some people,” says Nicolette, “is the idea that agile methods can be used with traditional development. We need to decouple the word ‘agile’ from ‘adaptive’ because quite often it is used in a traditional context.”

Scenario: A team wanting to optimize their working practices to stay on top of a bug queue.

Typically “you want to be somewhat predictable in your service time,” says Nicolette, “so that when a bug report comes in people have an idea of when they can expect to see it fixed.” To be able to calculate this “you can use empirical information from past performance with fixing bugs, and your mean cycle time will give you approximately how long it takes to fix one.”

However, “I like to use the Kanban method for these kinds of teams because it defines classes of service. You’ll find that every type of bug doesn’t take the same amount of time to fix,” cautions Nicolette. So “based on your history, pick out different categories. You can identify the characteristics of particular kinds of bug reports that tend to fall together, and you can track cycle times differently for each of those classes of service.”

Scenario: A CTO who wants to monitor how teams are performing and ensure code is of high quality.

Nicolette recommends in order to measure “how the teams are performing, you need measurements that are comparable across teams and comparable across projects.” To to this end, he recommends “tracking throughput, cycle time” and “process cycle efficiency” as “those roll up nicely.” However, he warns that “some other metrics don’t roll up so well” — “like agile metrics, particularly velocity, which is really different for each team”.

“The other question about code quality, I think should be a team responsibility. Then they can use metrics, but they don’t need to report them outside of the team. Usually, things that they can get out of static code analysis tools will help them spot potential quality issues, but I wouldn’t share very detailed things like static code analysis… outside the team, because then team members will feel like they’re going to get judged on that and they’ll start gaming the numbers.”

Read more interviews with some of the world’s best developers on * or follow us on or follow us on .*